Jul 07 14:10:05.310852 ip-10-0-140-100 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.745472    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750501    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750515    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750519    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750523    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750526    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750531    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750534    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.757192 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750536    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750539    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750542    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750546    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750549    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750551    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750555    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750558    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750561    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750564    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750566    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750573    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750577    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750580    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750583    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750586    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750589    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750592    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750595    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750598    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.759682 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750601    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750604    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750607    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750610    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750613    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750616    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750619    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750621    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750624    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750627    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750630    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750633    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750635    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750638    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750642    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750645    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750647    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750651    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750654    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.760379 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750657    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750659    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750662    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750665    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750668    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750671    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750674    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750676    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750679    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750682    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750685    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750688    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750692    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750697    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750700    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750703    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750706    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750708    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750711    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750714    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.761074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750717    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750720    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750722    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750725    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750730    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750733    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750737    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750740    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750743    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750747    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750750    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750753    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750756    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750759    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750763    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750766    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750768    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750771    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750774    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750777    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.761782 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750779    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750782    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750786    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750789    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750792    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750795    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750798    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750802    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750807    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750810    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750813    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750817    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.750820    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751698    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751711    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751715    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751719    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751722    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751725    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.762934 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751727    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751730    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751734    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751737    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751740    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751743    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751747    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751751    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751754    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751757    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751760    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751763    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751766    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751768    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751771    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751774    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751778    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751781    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751784    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751787    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.763626 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751790    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751792    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751795    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751798    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751801    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751804    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751807    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751812    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751815    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751818    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751821    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751824    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751828    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751831    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751834    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751837    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751840    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751843    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751845    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751848    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.765124 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751851    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751854    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751857    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751859    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751862    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751865    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751868    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751872    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751877    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751880    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751883    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751886    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751889    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751892    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751896    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751899    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751902    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751905    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751908    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.765830 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751911    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751917    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751921    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751924    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751927    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751930    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751933    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751936    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751939    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751941    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751944    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751947    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751949    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751952    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751955    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751957    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751960    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751962    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751965    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.766981 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751978    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751982    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751985    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751987    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751991    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751994    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751997    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.751999    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752002    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752005    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752008    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752011    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752013    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752016    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752458    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752462    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752466    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752469    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.767675 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752472    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752475    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752478    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752480    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752483    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752487    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752492    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752495    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752499    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752501    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752504    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752507    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752511    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752514    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752517    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752519    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752522    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752525    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752528    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752530    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.768825 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752533    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752536    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752539    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752541    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752544    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752549    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752553    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752557    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752560    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752563    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752566    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752569    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752572    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752574    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752577    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752580    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752582    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752586    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752590    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752593    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.769510 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752595    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752598    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752601    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752603    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752606    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752615    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752618    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752621    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752624    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752627    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752630    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752633    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752636    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752638    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752641    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752644    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752648    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752652    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752655    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752658    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.770901 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752662    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752665    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752668    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752671    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752674    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752677    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752680    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752683    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752686    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752689    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752691    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752694    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752697    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752699    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752702    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752705    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752707    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752712    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752715    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752717    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.771593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752720    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752723    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752725    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752729    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752732    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752735    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752737    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752740    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752743    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752745    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752748    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752751    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752753    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.752757    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752840    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752847    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752854    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752859    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752864    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752868    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 14:10:05.772717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752872    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752877    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752881    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752884    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752888    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752891    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752895    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752898    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752901    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752906    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752909    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752912    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752917    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752920    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752923    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752927    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752930    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752935    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752938    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752941    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752945    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752948    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752951    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752954    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 14:10:05.773388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752957    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752960    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752970    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752973    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752976    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752979    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752982    2563 flags.go:64] FLAG: --enable-server="true"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752985    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752991    2563 flags.go:64] FLAG: --event-burst="100"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752994    2563 flags.go:64] FLAG: --event-qps="50"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.752997    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753000    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753003    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753012    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753015    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753018    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753021    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753026    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753030    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753034    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753037    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753040    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753043    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753046    2563 flags.go:64] FLAG: --feature-gates=""
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753050    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 14:10:05.774309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753053    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753056    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753060    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753063    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753067    2563 flags.go:64] FLAG: --help="false"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753070    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753074    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753077    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753080    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753083    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753088    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753091    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753094    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753097    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753100    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753103    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753106    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753109    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753112    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753115    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753119    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753122    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753125    2563 flags.go:64] FLAG: --lock-file=""
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753128    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 14:10:05.775195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753132    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753137    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753143    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753146    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753150    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753153    2563 flags.go:64] FLAG: --logging-format="text"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753156    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753159    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753162    2563 flags.go:64] FLAG: --manifest-url=""
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753165    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753170    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753187    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753192    2563 flags.go:64] FLAG: --max-pods="110"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753195    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753198    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753201    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753204    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753207    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753211    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753215    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753222    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753226    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753229    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 14:10:05.776079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753232    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753235    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753241    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753244    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753248    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753251    2563 flags.go:64] FLAG: --port="10250"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753254    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753257    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c2d4bad10935ed1"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753261    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753266    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753269    2563 flags.go:64] FLAG: --register-node="true"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753273    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753277    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753280    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753284    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753287    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753291    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753294    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753297    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753300    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753303    2563 flags.go:64] FLAG: --runonce="false"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753306    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753309    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753312    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753315    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 14:10:05.776832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753319    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753322    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753325    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753329    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753332    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753336    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753338    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753341    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753345    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753348    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753351    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753356    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753359    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753362    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753366    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753369    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753373    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753377    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753382    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753385    2563 flags.go:64] FLAG: --v="2"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753389    2563 flags.go:64] FLAG: --version="false"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753394    2563 flags.go:64] FLAG: --vmodule=""
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753397    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.753401    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753523    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.777584 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753527    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753531    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753534    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753537    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753540    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753543    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753547    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753549    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753554    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753559    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753562    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753565    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753569    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753572    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753575    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753577    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753580    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753583    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753585    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.778339 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753588    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753590    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753593    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753596    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753600    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753602    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753607    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753609    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753612    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753615    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753618    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753621    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753623    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753626    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753629    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753632    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753634    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753637    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753640    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753643    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.779148 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753645    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753648    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753651    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753654    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753656    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753660    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753663    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753665    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753668    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753671    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753674    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753677    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753679    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753682    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753685    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753689    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753693    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753697    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753702    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753705    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.779776 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753708    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753711    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753714    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753717    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753720    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753723    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753726    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753728    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753731    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753734    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753737    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753739    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753742    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753745    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753747    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753750    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753753    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753756    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753758    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753762    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.780406 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753764    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753767    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753770    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753772    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753775    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753778    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753780    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753783    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753787    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753789    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753794    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753797    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753799    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753802    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753805    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753807    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753810    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753813    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.781027 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.753816    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.754724    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.762739    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.762761    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.762784    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.762790    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762863    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762868    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762874    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762877    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762880    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762883    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762888    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762891    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762894    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.781595 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762898    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762903    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762908    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762914    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762918    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762922    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762925    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762928    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762930    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762933    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762936    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762939    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762944    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762947    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762950    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762955    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762958    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762961    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762964    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.782074 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762968    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762972    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762974    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762977    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762982    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762985    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762988    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762994    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.762998    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763001    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763006    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763009    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763012    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763015    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763020    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763023    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763026    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763029    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763034    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.782714 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763037    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763039    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763042    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763049    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763052    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763057    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763060    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763063    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763066    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763071    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763074    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763077    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763079    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763084    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763087    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763090    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763093    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763098    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763101    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.886155 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763103    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.846986 ip-10-0-140-100 systemd[1]: Started Kubernetes Kubelet.
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763106    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763111    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763113    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763116    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763119    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763125    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763128    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763131    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763136    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763139    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763141    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763146    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763149    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763154    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763157    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763160    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763163    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763166    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763188    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.887839 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763193    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763199    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763202    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763206    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763211    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763214    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763217    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763220    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763225    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763228    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763231    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763236    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763239    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.763245    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763414    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:10:05.889105 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763419    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763422    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763428    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763431    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763435    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763440    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763443    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763446    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763451    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763454    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763457    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763460    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763468    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763471    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763476    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763479    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763482    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763484    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763491    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:10:05.889811 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763494    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763498    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763503    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763507    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763509    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763513    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763518    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763521    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763527    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763532    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763535    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763537    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763540    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763545    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763547    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763552    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763554    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763559    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763561    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763564    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:10:05.890502 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763567    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763572    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763575    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763578    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763581    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763587    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763591    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763593    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763598    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763601    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763604    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763608    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763611    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763614    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763617    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763619    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763622    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763625    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763628    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763631    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:10:05.891676 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763635    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763639    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763642    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763648    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763652    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763655    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763660    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763664    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763666    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763669    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763674    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763677    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763679    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763682    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763687    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763690    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763693    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763699    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763702    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:10:05.892351 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763705    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763708    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763711    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763718    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763721    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763724    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763729    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763732    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763735    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763738    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763746    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763749    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763751    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763754    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763759    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763762    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763767    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:10:05.893279 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:05.763772    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.763777    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.763911    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.778301    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.779907    2563 server.go:1019] "Starting client certificate rotation"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.780016    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.780049    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.782792    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.794200    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.812360    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.814953    2563 log.go:25] "Validated CRI v1 image API"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.815803    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 14:10:05.893913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.818342    2563 fs.go:135] Filesystem UUIDs: map[3fa5f9ea-f445-491a-86ee-826fbc477ffb:/dev/nvme0n1p4 4462abae-bffa-4416-87c6-12a75cff14ed:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 07 14:10:05.894583 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.818358    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 14:10:05.894583 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.822350    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.824639    2563 manager.go:216] Machine: {Timestamp:2026-07-07 14:10:05.82404307 +0000 UTC m=+0.380479429 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100929 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec265654684aac5f794ee401e40b46eb SystemUUID:ec265654-684a-ac5f-794e-e401e40b46eb BootID:681c6266-4dda-4d3d-ad96-fe001ae37f1c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:d1:13:c9:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:d1:13:c9:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:78:7d:52:cc:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.824749    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.824836    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.827508    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.827536    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.828758    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.828769    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.828778    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.829846    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.831556    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835277    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835400    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835431    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835447    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835471    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835484    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835497    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 14:10:05.895885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835939    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.835957    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.838213    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.840461    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.840490    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841773    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841791    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841801    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841811    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841823    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841829    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841838    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841844    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841851    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841857    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841866    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.841872    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.842560    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.842570    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.846097    2563 server.go:1268] "Started kubelet"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.846212    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.846328    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.846495    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.848791    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.848886    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.849166    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.849764    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.849764    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.851934    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.855625    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6vpd"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.864277    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6vpd"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.864857    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.864903    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.864970    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.865448    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.865450    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.865472    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.865585    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.865596    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.866538    2563 factory.go:55] Registering systemd factory
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.866766    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867030    2563 factory.go:153] Registering CRI-O factory
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867071    2563 factory.go:223] Registration of the crio container factory successfully
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867141    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867166    2563 factory.go:103] Registering Raw factory
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867183    2563 manager.go:1196] Started watching for new ooms in manager
Jul 07 14:10:05.896552 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.867586    2563 manager.go:349] Starting recovery of all containers
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.866683    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-100.ec2.internal.18c0073bcc0508de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-100.ec2.internal,UID:ip-10-0-140-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-100.ec2.internal,},FirstTimestamp:2026-07-07 14:10:05.846063326 +0000 UTC m=+0.402499684,LastTimestamp:2026-07-07 14:10:05.846063326 +0000 UTC m=+0.402499684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-100.ec2.internal,}"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.879120    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.886312    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-100.ec2.internal\" not found" node="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.888372    2563 manager.go:354] Recovery completed
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.893562    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.897929    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.897958    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:05.898013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.897972    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:05.898778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.898607    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 14:10:05.898778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.898615    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 14:10:05.898778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.898619    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:05.898778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.898633    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:10:05.899580 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899353    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:05.899580 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899380    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:05.899580 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899394    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:05.899847 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899728    2563 policy_none.go:49] "None policy: Start"
Jul 07 14:10:05.899847 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899741    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 14:10:05.899847 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.899752    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.900662    2563 policy_none.go:47] "Start"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.948804    2563 manager.go:337] "Starting Device Plugin manager"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.948989    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949004    2563 server.go:87] "Starting device plugin registration server"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949286    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949300    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949415    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949491    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:05.949500    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.950116    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 14:10:06.021982 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:05.950167    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.050297    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.051435    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.051470    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.051485    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.051524    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.061127    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.061149    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-100.ec2.internal\": node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.073006    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.081089    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.082320    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.082360    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.082380    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.082418    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 14:10:06.158914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.084609    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.173910    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.182668    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"]
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.182757    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.184542    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.184573    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.184587    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.185778    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.185899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.185931    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186556    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186563    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186588    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186601    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186609    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.186630    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.189097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.189123    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.191467    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.191496    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.191506    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.236097    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-100.ec2.internal\" not found" node="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.240985    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-100.ec2.internal\" not found" node="ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.266732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.266761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.266779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/044b16742602656ad17c3b9ae8672de3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-100.ec2.internal\" (UID: \"044b16742602656ad17c3b9ae8672de3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.296516 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.274998    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/044b16742602656ad17c3b9ae8672de3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-100.ec2.internal\" (UID: \"044b16742602656ad17c3b9ae8672de3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367839    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/044b16742602656ad17c3b9ae8672de3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-100.ec2.internal\" (UID: \"044b16742602656ad17c3b9ae8672de3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.367923    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fddc92aed329b23447bf1d854ba67ded-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal\" (UID: \"fddc92aed329b23447bf1d854ba67ded\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.405565 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.375385    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.531237 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.476209    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.541036 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.541005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.556786 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.556762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:06.576776 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.576743    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.677570 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.677532    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.778218 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.778188    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.779373 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.779357    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 14:10:06.779527 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.779509    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:10:06.779581 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.779558    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:10:06.866195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.866087    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:05:06 +0000 UTC" deadline="2028-01-27 20:39:32.474583215 +0000 UTC"
Jul 07 14:10:06.866195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.866142    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13662h29m25.608443904s"
Jul 07 14:10:06.878553 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.878532    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.975912 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.975888    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:06.979464 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:06.979424    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-100.ec2.internal\" not found"
Jul 07 14:10:06.979761 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:06.979746    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:07.069236 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.069207    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:07.081883 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.081863    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:10:07.083005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.082992    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal"
Jul 07 14:10:07.094268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.094249    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:10:07.259942 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:07.259906    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddc92aed329b23447bf1d854ba67ded.slice/crio-47b1b6ee2633b44d740b7042b7f69406fd8f384b6b27e6cababb54d17ec4e9a6 WatchSource:0}: Error finding container 47b1b6ee2633b44d740b7042b7f69406fd8f384b6b27e6cababb54d17ec4e9a6: Status 404 returned error can't find the container with id 47b1b6ee2633b44d740b7042b7f69406fd8f384b6b27e6cababb54d17ec4e9a6
Jul 07 14:10:07.266366 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.266348    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:10:07.836976 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.836942    2563 apiserver.go:52] "Watching apiserver"
Jul 07 14:10:07.846244 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.846219    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 14:10:07.847566 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.847509    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-z7mkx","openshift-network-diagnostics/network-check-target-hppq5","kube-system/global-pull-secret-syncer-vjsqs","openshift-cluster-node-tuning-operator/tuned-d7fsq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal","openshift-multus/network-metrics-daemon-hp9vf","openshift-network-operator/iptables-alerter-9qn9s","openshift-ovn-kubernetes/ovnkube-node-9dnd7","kube-system/konnectivity-agent-bkxrc","kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5","openshift-dns/node-resolver-h5x7b","openshift-image-registry/node-ca-k8qgz","openshift-multus/multus-additional-cni-plugins-xk4q9"]
Jul 07 14:10:07.849037 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.849016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.850193 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.850173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.851448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.851430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.851592 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.851503    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:07.852282 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.852075    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.852282 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.852122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 14:10:07.852434 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.852277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-28dsf\""
Jul 07 14:10:07.852434 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.852333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.852714 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.852692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.853390 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.853365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.855207 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.854246    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.855207 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.854297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 14:10:07.855207 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.854826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.855207 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.855023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfzz8\""
Jul 07 14:10:07.857080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.856038    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gj5rl\""
Jul 07 14:10:07.857080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.856209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.857080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.856480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.857283 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.857247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.857413 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.857385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:07.858689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.858281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.858689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.858347    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.860041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.859487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 14:10:07.860041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.859681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.860041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.859854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwtt6\""
Jul 07 14:10:07.860041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.859682    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 14:10:07.860041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.859984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 14:10:07.860766 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.860749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 14:10:07.860936 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.860915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.861754 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.861736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 14:10:07.861851 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.861773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6vdq6\""
Jul 07 14:10:07.862872 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.862853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:07.862978 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.862923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:07.863755 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.863722    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.863755 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.863747    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.863928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.863855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 14:10:07.863928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.863863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hpgwz\""
Jul 07 14:10:07.864403 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.864385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.866151 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.865938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.866260 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.866201    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 14:10:07.867349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.867300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vsfc9\""
Jul 07 14:10:07.867349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.867331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.867507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.867368    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 14:10:07.867507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.867450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.868656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.867893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.868656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.868538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 14:10:07.868656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.868552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 14:10:07.868847 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.868750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 14:10:07.868847 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.868779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 14:10:07.868951 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.868901    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tb2ss\""
Jul 07 14:10:07.870656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.870638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 14:10:07.870887 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.870862    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 14:10:07.870961 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.870862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dlvgt\""
Jul 07 14:10:07.877783 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-serviceca\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.877894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-bin\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.877894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.877894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlbv\" (UniqueName: \"kubernetes.io/projected/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-kube-api-access-ltlbv\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-var-lib-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.877973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-k8s-cni-cncf-io\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jnr7\" (UniqueName: \"kubernetes.io/projected/a779b2ca-332b-46f4-be32-69bddff5607b-kube-api-access-5jnr7\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-kubernetes\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-binary-copy\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.878077 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-config\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-hostroot\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-daemon-config\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-etc-kubernetes\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhsn\" (UniqueName: \"kubernetes.io/projected/fa830b74-639c-4faa-9cdf-9a4f19642394-kube-api-access-rwhsn\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-modprobe-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.878370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwn4\" (UniqueName: \"kubernetes.io/projected/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-kube-api-access-rmwn4\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa830b74-639c-4faa-9cdf-9a4f19642394-host-slash\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-sys-fs\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-hosts-file\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-registration-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-var-lib-kubelet\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ww2\" (UniqueName: \"kubernetes.io/projected/d527cbd4-42a4-4d5e-ade4-8d730815af83-kube-api-access-k4ww2\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-system-cni-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.878746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh6j\" (UniqueName: \"kubernetes.io/projected/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-kube-api-access-phh6j\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-sys\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g959f\" (UniqueName: \"kubernetes.io/projected/6cc543ad-56d9-474e-aac9-21e6f2e00224-kube-api-access-g959f\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd4c208-30a8-4b72-b694-87fcfd85d46f-agent-certs\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4c208-30a8-4b72-b694-87fcfd85d46f-konnectivity-ca\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-netns\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-netns\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-multus\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.878981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-dbus\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-systemd-units\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-kubelet\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-socket-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysconfig\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-systemd\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.879340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-lib-modules\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa830b74-639c-4faa-9cdf-9a4f19642394-iptables-alerter-script\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-kubelet-config\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbqlb\" (UniqueName: \"kubernetes.io/projected/00f7d4fa-1d46-4d30-964e-ab7dac77df46-kube-api-access-zbqlb\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-slash\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-node-log\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-os-release\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-tmp-dir\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-run\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-os-release\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-system-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-bin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-conf\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-host\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.880145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-env-overrides\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-socket-dir-parent\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-conf-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-tmp\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-log-socket\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-script-lib\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-cni-binary-copy\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-multus-certs\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwkd\" (UniqueName: \"kubernetes.io/projected/bc634bdd-0d3d-4b71-90f5-5449569cac27-kube-api-access-2qwkd\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.879987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-host\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-kubelet\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-systemd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-cnibin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-device-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-cnibin\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.880828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-etc-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.881500 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-ovn\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.881500 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-netd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.881500 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.881500 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.880276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-tuned\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.882402 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.882381    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:10:07.911240 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.911218    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rpbzb"
Jul 07 14:10:07.919656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.919560    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rpbzb"
Jul 07 14:10:07.966550 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.966521    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 14:10:07.980520 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwn4\" (UniqueName: \"kubernetes.io/projected/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-kube-api-access-rmwn4\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa830b74-639c-4faa-9cdf-9a4f19642394-host-slash\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-sys-fs\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.980662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-hosts-file\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-registration-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-var-lib-kubelet\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ww2\" (UniqueName: \"kubernetes.io/projected/d527cbd4-42a4-4d5e-ade4-8d730815af83-kube-api-access-k4ww2\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-system-cni-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980800    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-system-cni-dir\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh6j\" (UniqueName: \"kubernetes.io/projected/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-kube-api-access-phh6j\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-sys\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980854    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa830b74-639c-4faa-9cdf-9a4f19642394-host-slash\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g959f\" (UniqueName: \"kubernetes.io/projected/6cc543ad-56d9-474e-aac9-21e6f2e00224-kube-api-access-g959f\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980902    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-sys-fs\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd4c208-30a8-4b72-b694-87fcfd85d46f-agent-certs\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980931    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-registration-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4c208-30a8-4b72-b694-87fcfd85d46f-konnectivity-ca\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.980988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.980988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-netns\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-netns\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-multus\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-dbus\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-systemd-units\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981187    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-systemd-units\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981223    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-sys\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.981283    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981351    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-run-netns\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.981360    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:08.48133778 +0000 UTC m=+3.037774141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981405    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-hosts-file\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981431    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-netns\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981465    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-multus\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981464    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4c208-30a8-4b72-b694-87fcfd85d46f-konnectivity-ca\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-kubelet\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981524    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-var-lib-kubelet\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981524    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 14:10:07.981866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-socket-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-kubelet\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysconfig\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981650    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysconfig\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-systemd\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-socket-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-lib-modules\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981699    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-systemd\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa830b74-639c-4faa-9cdf-9a4f19642394-iptables-alerter-script\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-dbus\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-kubelet-config\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbqlb\" (UniqueName: \"kubernetes.io/projected/00f7d4fa-1d46-4d30-964e-ab7dac77df46-kube-api-access-zbqlb\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-slash\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-node-log\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981813    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-lib-modules\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981814    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/565c84fb-b78f-4739-bcd8-216275ac3643-kubelet-config\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-os-release\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-tmp-dir\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.982564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981862    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-slash\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-run\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-os-release\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-os-release\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981938    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-node-log\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-system-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.981979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-bin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-conf\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-host\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982087    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982092    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-env-overrides\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-socket-dir-parent\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982142    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-run\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.983454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-conf-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-system-cni-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-var-lib-cni-bin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982190    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-os-release\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-tmp\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982241    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-socket-dir-parent\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982281    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-sysctl-conf\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982292    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-conf-dir\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-log-socket\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-script-lib\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982342    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-log-socket\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982305    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa830b74-639c-4faa-9cdf-9a4f19642394-iptables-alerter-script\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-cni-binary-copy\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-multus-certs\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwkd\" (UniqueName: \"kubernetes.io/projected/bc634bdd-0d3d-4b71-90f5-5449569cac27-kube-api-access-2qwkd\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.982426    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.984349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-host\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982500    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-multus-certs\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.982505    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:08.482488723 +0000 UTC m=+3.038925084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982241    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-tmp-dir\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982599    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982649    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-host\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982686    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982294    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-host\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-kubelet\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-systemd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-cnibin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982771    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-env-overrides\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982790    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-kubelet\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982801    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-cnibin\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982819    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-systemd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-device-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-cnibin\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.985275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-etc-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-ovn\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982930    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-etc-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-netd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982952    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-device-dir\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982968    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-netd\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.982975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-tuned\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983010    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cc543ad-56d9-474e-aac9-21e6f2e00224-cnibin\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-serviceca\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-bin\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983059    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-cni-binary-copy\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlbv\" (UniqueName: \"kubernetes.io/projected/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-kube-api-access-ltlbv\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983101    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a779b2ca-332b-46f4-be32-69bddff5607b-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-var-lib-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986106 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-k8s-cni-cncf-io\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnr7\" (UniqueName: \"kubernetes.io/projected/a779b2ca-332b-46f4-be32-69bddff5607b-kube-api-access-5jnr7\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-var-lib-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983021    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-ovn\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-kubernetes\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983288    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-script-lib\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983296    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-kubernetes\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-run-openvswitch\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983355    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-host-run-k8s-cni-cncf-io\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983361    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-cni-bin\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-binary-copy\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-config\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-hostroot\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-daemon-config\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-etc-kubernetes\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhsn\" (UniqueName: \"kubernetes.io/projected/fa830b74-639c-4faa-9cdf-9a4f19642394-kube-api-access-rwhsn\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.986841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-hostroot\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-modprobe-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983650    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-serviceca\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983679    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983718    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc634bdd-0d3d-4b71-90f5-5449569cac27-etc-kubernetes\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983722    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-modprobe-d\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.983974    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cc543ad-56d9-474e-aac9-21e6f2e00224-cni-binary-copy\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.984160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovnkube-config\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.984361    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc634bdd-0d3d-4b71-90f5-5449569cac27-multus-daemon-config\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.986566    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-etc-tuned\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.986599    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.986613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d527cbd4-42a4-4d5e-ade4-8d730815af83-tmp\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.987614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.986738    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9fd4c208-30a8-4b72-b694-87fcfd85d46f-agent-certs\") pod \"konnectivity-agent-bkxrc\" (UID: \"9fd4c208-30a8-4b72-b694-87fcfd85d46f\") " pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:07.994959 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.994933    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:07.995074 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.994965    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:07.995074 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.994979    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:07.995074 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:07.995067    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:08.495049103 +0000 UTC m=+3.051485447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:07.995482 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.995410    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ww2\" (UniqueName: \"kubernetes.io/projected/d527cbd4-42a4-4d5e-ade4-8d730815af83-kube-api-access-k4ww2\") pod \"tuned-d7fsq\" (UID: \"d527cbd4-42a4-4d5e-ade4-8d730815af83\") " pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:07.996036 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.995897    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g959f\" (UniqueName: \"kubernetes.io/projected/6cc543ad-56d9-474e-aac9-21e6f2e00224-kube-api-access-g959f\") pod \"multus-additional-cni-plugins-xk4q9\" (UID: \"6cc543ad-56d9-474e-aac9-21e6f2e00224\") " pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:07.996227 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.996164    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwkd\" (UniqueName: \"kubernetes.io/projected/bc634bdd-0d3d-4b71-90f5-5449569cac27-kube-api-access-2qwkd\") pod \"multus-z7mkx\" (UID: \"bc634bdd-0d3d-4b71-90f5-5449569cac27\") " pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:07.996415 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.996357    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwn4\" (UniqueName: \"kubernetes.io/projected/cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b-kube-api-access-rmwn4\") pod \"node-resolver-h5x7b\" (UID: \"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b\") " pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:07.996587 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.996518    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhsn\" (UniqueName: \"kubernetes.io/projected/fa830b74-639c-4faa-9cdf-9a4f19642394-kube-api-access-rwhsn\") pod \"iptables-alerter-9qn9s\" (UID: \"fa830b74-639c-4faa-9cdf-9a4f19642394\") " pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:07.996941 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.996922    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh6j\" (UniqueName: \"kubernetes.io/projected/487a3a9c-267e-4ac8-82c5-00a1de9a0d9c-kube-api-access-phh6j\") pod \"ovnkube-node-9dnd7\" (UID: \"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:07.997898 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.997851    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnr7\" (UniqueName: \"kubernetes.io/projected/a779b2ca-332b-46f4-be32-69bddff5607b-kube-api-access-5jnr7\") pod \"aws-ebs-csi-driver-node-r4tn5\" (UID: \"a779b2ca-332b-46f4-be32-69bddff5607b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:07.998310 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.998287    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbqlb\" (UniqueName: \"kubernetes.io/projected/00f7d4fa-1d46-4d30-964e-ab7dac77df46-kube-api-access-zbqlb\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:07.998647 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:07.998625    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlbv\" (UniqueName: \"kubernetes.io/projected/c85353ad-cb20-4ae6-8bf0-36406ff4c93e-kube-api-access-ltlbv\") pod \"node-ca-k8qgz\" (UID: \"c85353ad-cb20-4ae6-8bf0-36406ff4c93e\") " pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:08.086268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.086215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal" event={"ID":"fddc92aed329b23447bf1d854ba67ded","Type":"ContainerStarted","Data":"47b1b6ee2633b44d740b7042b7f69406fd8f384b6b27e6cababb54d17ec4e9a6"}
Jul 07 14:10:08.126912 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.126847    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:08.137897 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.137871    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:10:08.163492 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.163470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qn9s"
Jul 07 14:10:08.171645 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.171625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5"
Jul 07 14:10:08.175602 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.175563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa830b74_639c_4faa_9cdf_9a4f19642394.slice/crio-242a49d668404c3631d90409923bd85190a815d68c34609b367514abf7a1a5a8 WatchSource:0}: Error finding container 242a49d668404c3631d90409923bd85190a815d68c34609b367514abf7a1a5a8: Status 404 returned error can't find the container with id 242a49d668404c3631d90409923bd85190a815d68c34609b367514abf7a1a5a8
Jul 07 14:10:08.180370 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.180349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda779b2ca_332b_46f4_be32_69bddff5607b.slice/crio-c94f2852d8da81a6e3382efc206e74720b6e50a74a4e6a616b96c16364f10e6a WatchSource:0}: Error finding container c94f2852d8da81a6e3382efc206e74720b6e50a74a4e6a616b96c16364f10e6a: Status 404 returned error can't find the container with id c94f2852d8da81a6e3382efc206e74720b6e50a74a4e6a616b96c16364f10e6a
Jul 07 14:10:08.181791 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.181773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7fsq"
Jul 07 14:10:08.186404 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.186383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xk4q9"
Jul 07 14:10:08.191073 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.191054    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd527cbd4_42a4_4d5e_ade4_8d730815af83.slice/crio-bfb6efd92c0449d6a5c5ca478c21f60a70ca5da5e95bec1703c0442f83d9e954 WatchSource:0}: Error finding container bfb6efd92c0449d6a5c5ca478c21f60a70ca5da5e95bec1703c0442f83d9e954: Status 404 returned error can't find the container with id bfb6efd92c0449d6a5c5ca478c21f60a70ca5da5e95bec1703c0442f83d9e954
Jul 07 14:10:08.193628 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.193598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:08.195691 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.195670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc543ad_56d9_474e_aac9_21e6f2e00224.slice/crio-fb5f1fa1da15d1d556c0a7d2591d9147baead99e3a05e7b1f081842d56a8ec62 WatchSource:0}: Error finding container fb5f1fa1da15d1d556c0a7d2591d9147baead99e3a05e7b1f081842d56a8ec62: Status 404 returned error can't find the container with id fb5f1fa1da15d1d556c0a7d2591d9147baead99e3a05e7b1f081842d56a8ec62
Jul 07 14:10:08.200865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.200837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k8qgz"
Jul 07 14:10:08.201209 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.201186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd4c208_30a8_4b72_b694_87fcfd85d46f.slice/crio-4ff6ae742624efeed257e16b8eda790ca86c69fc7edab71fdbac3c34262a01d3 WatchSource:0}: Error finding container 4ff6ae742624efeed257e16b8eda790ca86c69fc7edab71fdbac3c34262a01d3: Status 404 returned error can't find the container with id 4ff6ae742624efeed257e16b8eda790ca86c69fc7edab71fdbac3c34262a01d3
Jul 07 14:10:08.209090 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.209070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:08.210654 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.210631    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85353ad_cb20_4ae6_8bf0_36406ff4c93e.slice/crio-fce257eef27fb7fc36e03025ec42e49704283ff82fcad76c7705e5c33e47efbc WatchSource:0}: Error finding container fce257eef27fb7fc36e03025ec42e49704283ff82fcad76c7705e5c33e47efbc: Status 404 returned error can't find the container with id fce257eef27fb7fc36e03025ec42e49704283ff82fcad76c7705e5c33e47efbc
Jul 07 14:10:08.216139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.216115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7mkx"
Jul 07 14:10:08.222864 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.222847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h5x7b"
Jul 07 14:10:08.223697 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.223672    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc634bdd_0d3d_4b71_90f5_5449569cac27.slice/crio-7c9559c0c6c1f74b7354cda075badff39f089ce65d7d60ce7258c8e4614af741 WatchSource:0}: Error finding container 7c9559c0c6c1f74b7354cda075badff39f089ce65d7d60ce7258c8e4614af741: Status 404 returned error can't find the container with id 7c9559c0c6c1f74b7354cda075badff39f089ce65d7d60ce7258c8e4614af741
Jul 07 14:10:08.230415 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.230391    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbad1b4d_6888_4aeb_a6bb_7cdcf59f8e5b.slice/crio-e4a6267c56a6fd528bd900954a6a05f988d498545a298098be3f81ebed29767e WatchSource:0}: Error finding container e4a6267c56a6fd528bd900954a6a05f988d498545a298098be3f81ebed29767e: Status 404 returned error can't find the container with id e4a6267c56a6fd528bd900954a6a05f988d498545a298098be3f81ebed29767e
Jul 07 14:10:08.487341 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.487253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:08.487341 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.487327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:08.487574 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.487442    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:08.487574 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.487467    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:08.487574 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.487523    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:09.487504394 +0000 UTC m=+4.043940741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:08.487574 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.487543    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:09.487533011 +0000 UTC m=+4.043969355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:08.588690 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.588644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:08.588828 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.588784    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:08.588828 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.588802    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:08.588828 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.588815    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:08.589009 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:08.588921    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:09.588900708 +0000 UTC m=+4.145337069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:08.590434 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:10:08.590405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044b16742602656ad17c3b9ae8672de3.slice/crio-be4fff4ce84059d33d497b2001b81d012ec8e53f4b9cb6db9c8ffadc378dc9c7 WatchSource:0}: Error finding container be4fff4ce84059d33d497b2001b81d012ec8e53f4b9cb6db9c8ffadc378dc9c7: Status 404 returned error can't find the container with id be4fff4ce84059d33d497b2001b81d012ec8e53f4b9cb6db9c8ffadc378dc9c7
Jul 07 14:10:08.923429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.923213    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:05:08 +0000 UTC" deadline="2027-12-22 03:24:20.193140211 +0000 UTC"
Jul 07 14:10:08.923429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:08.923266    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12781h14m11.269879427s"
Jul 07 14:10:09.083514 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.083311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:09.083672 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.083627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:09.090547 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.090490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7fsq" event={"ID":"d527cbd4-42a4-4d5e-ade4-8d730815af83","Type":"ContainerStarted","Data":"bfb6efd92c0449d6a5c5ca478c21f60a70ca5da5e95bec1703c0442f83d9e954"}
Jul 07 14:10:09.096100 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.096072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5" event={"ID":"a779b2ca-332b-46f4-be32-69bddff5607b","Type":"ContainerStarted","Data":"c94f2852d8da81a6e3382efc206e74720b6e50a74a4e6a616b96c16364f10e6a"}
Jul 07 14:10:09.099285 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.099257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"5f3ed386e7921092e0903216107586b5ca298123d9f0c24c55a153d245167984"}
Jul 07 14:10:09.104772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.104708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k8qgz" event={"ID":"c85353ad-cb20-4ae6-8bf0-36406ff4c93e","Type":"ContainerStarted","Data":"fce257eef27fb7fc36e03025ec42e49704283ff82fcad76c7705e5c33e47efbc"}
Jul 07 14:10:09.106476 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.106408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qn9s" event={"ID":"fa830b74-639c-4faa-9cdf-9a4f19642394","Type":"ContainerStarted","Data":"242a49d668404c3631d90409923bd85190a815d68c34609b367514abf7a1a5a8"}
Jul 07 14:10:09.111285 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.110728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal" event={"ID":"fddc92aed329b23447bf1d854ba67ded","Type":"ContainerStarted","Data":"91cd8efddd5c7c350fc217a47e9a84b53e47a113e97f452899ab77f9c9630d0d"}
Jul 07 14:10:09.113388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.113327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal" event={"ID":"044b16742602656ad17c3b9ae8672de3","Type":"ContainerStarted","Data":"be4fff4ce84059d33d497b2001b81d012ec8e53f4b9cb6db9c8ffadc378dc9c7"}
Jul 07 14:10:09.115019 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.114994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h5x7b" event={"ID":"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b","Type":"ContainerStarted","Data":"e4a6267c56a6fd528bd900954a6a05f988d498545a298098be3f81ebed29767e"}
Jul 07 14:10:09.124991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.123924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7mkx" event={"ID":"bc634bdd-0d3d-4b71-90f5-5449569cac27","Type":"ContainerStarted","Data":"7c9559c0c6c1f74b7354cda075badff39f089ce65d7d60ce7258c8e4614af741"}
Jul 07 14:10:09.127063 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.127026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bkxrc" event={"ID":"9fd4c208-30a8-4b72-b694-87fcfd85d46f","Type":"ContainerStarted","Data":"4ff6ae742624efeed257e16b8eda790ca86c69fc7edab71fdbac3c34262a01d3"}
Jul 07 14:10:09.133396 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.133153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerStarted","Data":"fb5f1fa1da15d1d556c0a7d2591d9147baead99e3a05e7b1f081842d56a8ec62"}
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.495169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.495266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.495432    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.495500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:11.495478605 +0000 UTC m=+6.051914952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.495937    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:09.496037 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.496003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:11.49598634 +0000 UTC m=+6.052422689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:09.596092 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.596057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:09.596305 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.596291    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:09.596390 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.596318    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:09.596390 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.596357    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:09.596502 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:09.596416    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:11.596396953 +0000 UTC m=+6.152833330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:09.924421 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.924288    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:05:08 +0000 UTC" deadline="2028-02-09 12:08:57.201870726 +0000 UTC"
Jul 07 14:10:09.924421 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:09.924351    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13965h58m47.277523527s"
Jul 07 14:10:10.086640 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:10.085936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:10.086640 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:10.086061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:10.086640 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:10.086468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:10.086640 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:10.086579    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:10.148538 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:10.148370    2563 generic.go:356] "Generic (PLEG): container finished" podID="fddc92aed329b23447bf1d854ba67ded" containerID="91cd8efddd5c7c350fc217a47e9a84b53e47a113e97f452899ab77f9c9630d0d" exitCode=0
Jul 07 14:10:10.148538 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:10.148418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal" event={"ID":"fddc92aed329b23447bf1d854ba67ded","Type":"ContainerDied","Data":"91cd8efddd5c7c350fc217a47e9a84b53e47a113e97f452899ab77f9c9630d0d"}
Jul 07 14:10:11.082945 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:11.082907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:11.083467 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.083050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:11.516397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:11.516473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.516548    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.516612    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.516639    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.516614813 +0000 UTC m=+10.073051166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:11.517032 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.516664    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.516648256 +0000 UTC m=+10.073084606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:11.617553 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:11.616887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:11.617553 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.617116    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:11.617553 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.617155    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:11.617553 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.617172    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:11.617553 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:11.617237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.617215924 +0000 UTC m=+10.173652292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:12.084055 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:12.084021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:12.084570 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:12.084186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:12.084648 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:12.084599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:12.084781 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:12.084713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:13.083164 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:13.082702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:13.083164 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:13.082840    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:14.083622 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:14.083547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:14.084064 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:14.083701    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:14.084155 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:14.084089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:14.084240 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:14.084210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:15.083094 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:15.083055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:15.083272 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.083194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:15.550964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:15.550744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:15.550964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:15.550832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:15.551462 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.550966    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:15.551462 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.550970    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:15.551462 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.551046    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:23.551025832 +0000 UTC m=+18.107462197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:15.551462 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.551066    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:23.551053941 +0000 UTC m=+18.107490287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:15.651815 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:15.651760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:15.651979 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.651956    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:15.652053 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.651985    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:15.652053 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.651999    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:15.652165 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:15.652059    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:23.652040207 +0000 UTC m=+18.208476563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:16.084116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:16.084073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:16.084300 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:16.084199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:16.084383 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:16.084306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:16.084456 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:16.084435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:17.083300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:17.083266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:17.083752 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:17.083395    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:18.084433 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:18.084339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:18.084890 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:18.084474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:18.084890 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:18.084796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:18.085009 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:18.084896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:19.082927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:19.082892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:19.083113 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:19.083035    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:20.083203 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:20.083123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:20.083203 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:20.083172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:20.083689 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:20.083267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:20.083689 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:20.083403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:21.082833 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:21.082803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:21.082979 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:21.082914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:22.082841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:22.082689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:22.082841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:22.082698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:22.082841 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:22.082806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:22.083350 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:22.082891    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:23.083027 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:23.082993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:23.083538 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.083115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:23.609412 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:23.609382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:23.609590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:23.609445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:23.609590 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.609492    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:23.609590 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.609551    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:23.609590 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.609565    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.609544564 +0000 UTC m=+34.165980926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:23.609755 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.609600    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.609584697 +0000 UTC m=+34.166021039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:23.710244 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:23.710212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:23.710387 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.710329    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:23.710387 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.710345    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:23.710387 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.710354    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:23.710479 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:23.710408    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.710391785 +0000 UTC m=+34.266828127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:24.083539 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:24.083500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:24.083882 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:24.083618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:24.083882 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:24.083650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:24.083882 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:24.083700    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:25.083309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:25.083264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:25.083479 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:25.083403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:26.083386 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:26.083349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:26.083856 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:26.083457    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:26.083856 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:26.083497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:26.083856 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:26.083592    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:27.082975 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:27.082945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:27.083159 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:27.083057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:28.082775 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:28.082743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:28.083228 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:28.082744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:28.083228 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:28.082958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:28.083228 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:28.082999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:29.083590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.083229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:29.084339 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:29.083667    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:29.195976 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.195894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal" event={"ID":"fddc92aed329b23447bf1d854ba67ded","Type":"ContainerStarted","Data":"f795f8069a925d9d16e94df551b598debe47a8459ba020890d637362c00107a8"}
Jul 07 14:10:29.197524 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.197490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal" event={"ID":"044b16742602656ad17c3b9ae8672de3","Type":"ContainerStarted","Data":"077e1df0c74208bc5794ca8596449ced3a89ae868439458b8cc3b3084ca83dc2"}
Jul 07 14:10:29.198991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.198965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h5x7b" event={"ID":"cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b","Type":"ContainerStarted","Data":"d5e8c0859710bf5239aa4601b4df1e84a466dab9c1b0048d5c0ca832ece0e39f"}
Jul 07 14:10:29.200688 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.200659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7mkx" event={"ID":"bc634bdd-0d3d-4b71-90f5-5449569cac27","Type":"ContainerStarted","Data":"c7939b0cf68d6a6b0c33153c20515e608426edf9dfb8cc413dbd30b11f8a37fc"}
Jul 07 14:10:29.202164 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.202130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bkxrc" event={"ID":"9fd4c208-30a8-4b72-b694-87fcfd85d46f","Type":"ContainerStarted","Data":"bf8f8ee5121166aaa9ed9828ffaae6ce431af03fb86000fe8be15fe57f5352ac"}
Jul 07 14:10:29.203824 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.203798    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="0da35b73f45ff4e06dbbe09d3daea825ccb74b127e61035578a6759ed298369e" exitCode=0
Jul 07 14:10:29.203933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.203849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"0da35b73f45ff4e06dbbe09d3daea825ccb74b127e61035578a6759ed298369e"}
Jul 07 14:10:29.205534 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.205502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7fsq" event={"ID":"d527cbd4-42a4-4d5e-ade4-8d730815af83","Type":"ContainerStarted","Data":"f4f46714f298f662095c5c7451ddf3b993add0e53ef495af6d5777741203ec66"}
Jul 07 14:10:29.207633 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.207607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5" event={"ID":"a779b2ca-332b-46f4-be32-69bddff5607b","Type":"ContainerStarted","Data":"f0a4fa1c9d1de15f33021b7d6428e31f4dd652c29def753a72e27fa2297ffa8f"}
Jul 07 14:10:29.211211 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"f259b85b97bf33cb5db8c70375c47c5bb65104219946433b375852a6a221b940"}
Jul 07 14:10:29.211307 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"1f384cf23acd5ab22d4c390e48e7008e4ff292693ee81a8a5cf439d5aed65f1e"}
Jul 07 14:10:29.211369 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"d4b3961a6da1f78eda734693a5ff35a1e90d97ea7b50f69d23ef4020ab288961"}
Jul 07 14:10:29.211369 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"97d6dacb6c678a23bee427cdb2cfa27e829647d8dc52cca654ba6ccbdb057805"}
Jul 07 14:10:29.211455 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"e571aff7b05ab3d165fabb972189288a6851630bacd648e3f303a8f0160485f8"}
Jul 07 14:10:29.211455 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.211447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"440e2f11cfebcb46ff2d15c776192caded4b730b5bf9e2fb6617d03c44c01bdb"}
Jul 07 14:10:29.212676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.212658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k8qgz" event={"ID":"c85353ad-cb20-4ae6-8bf0-36406ff4c93e","Type":"ContainerStarted","Data":"40a8b38745e443abb8e99869b4b3e4498f1bcbba663c9fe5964d9bcfec4ad268"}
Jul 07 14:10:29.215160 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.215117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-100.ec2.internal" podStartSLOduration=22.215083886 podStartE2EDuration="22.215083886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.214575965 +0000 UTC m=+23.771012330" watchObservedRunningTime="2026-07-07 14:10:29.215083886 +0000 UTC m=+23.771520252"
Jul 07 14:10:29.232967 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.232922    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d7fsq" podStartSLOduration=3.2450599159999998 podStartE2EDuration="23.232906848s" totalImagesPullingTime="19.987846932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.232340029 +0000 UTC m=+23.788776396" watchObservedRunningTime="2026-07-07 14:10:29.232906848 +0000 UTC m=+23.789343209"
Jul 07 14:10:29.275538 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.275495    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h5x7b" podStartSLOduration=3.326833069 podStartE2EDuration="23.275478069s" totalImagesPullingTime="19.948645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.275157314 +0000 UTC m=+23.831593679" watchObservedRunningTime="2026-07-07 14:10:29.275478069 +0000 UTC m=+23.831914432"
Jul 07 14:10:29.292893 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.292852    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-100.ec2.internal" podStartSLOduration=22.292840258 podStartE2EDuration="22.292840258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.292341574 +0000 UTC m=+23.848777937" watchObservedRunningTime="2026-07-07 14:10:29.292840258 +0000 UTC m=+23.849276621"
Jul 07 14:10:29.308108 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.308061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bkxrc" podStartSLOduration=3.329886687 podStartE2EDuration="23.308050881s" totalImagesPullingTime="19.978164194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.307999524 +0000 UTC m=+23.864435888" watchObservedRunningTime="2026-07-07 14:10:29.308050881 +0000 UTC m=+23.864487246"
Jul 07 14:10:29.323357 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.323300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k8qgz" podStartSLOduration=3.351339019 podStartE2EDuration="23.323289028s" totalImagesPullingTime="19.971950009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.322681627 +0000 UTC m=+23.879117985" watchObservedRunningTime="2026-07-07 14:10:29.323289028 +0000 UTC m=+23.879725392"
Jul 07 14:10:29.492939 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.492909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:29.627876 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.627837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:29.749921 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.749900    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 14:10:29.968133 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.968024    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:10:29.749916088Z","UUID":"6c0ba9a0-a2ac-4c72-a947-039e9f16cd87","Handler":null,"Name":"","Endpoint":""}
Jul 07 14:10:29.969419 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.969400    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 14:10:29.969536 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:29.969436    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 14:10:30.082953 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.082872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:30.082953 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.082914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:30.083173 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:30.083013    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:30.083173 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:30.083160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:30.215701 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.215664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qn9s" event={"ID":"fa830b74-639c-4faa-9cdf-9a4f19642394","Type":"ContainerStarted","Data":"47f473d0a3592fb18bf6e206f6ffe49f000999def95a44204c472007dae3eba3"}
Jul 07 14:10:30.217444 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.217419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5" event={"ID":"a779b2ca-332b-46f4-be32-69bddff5607b","Type":"ContainerStarted","Data":"5ef0561c0c2ac80652cdaf2247572d4eb99f155b71fadf1ad5e4a9a284af1aaf"}
Jul 07 14:10:30.232438 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.232398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z7mkx" podStartSLOduration=4.197114988 podStartE2EDuration="24.232385937s" totalImagesPullingTime="20.035270949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:29.341447783 +0000 UTC m=+23.897884148" watchObservedRunningTime="2026-07-07 14:10:30.232385937 +0000 UTC m=+24.788822301"
Jul 07 14:10:30.232698 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.232679    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9qn9s" podStartSLOduration=4.221791309 podStartE2EDuration="24.232670763s" totalImagesPullingTime="20.010879454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:30.232142228 +0000 UTC m=+24.788578592" watchObservedRunningTime="2026-07-07 14:10:30.232670763 +0000 UTC m=+24.789107127"
Jul 07 14:10:30.369716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.369613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:30.469277 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.469238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:30.769445 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.769372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:30.938692 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:30.938658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:31.056763 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.056700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:31.082894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.082865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:31.083038 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:31.082991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:31.191609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.191564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:31.224320 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.224287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"396183ec1d9616faa03d496995841f6f906d40df130ac9cf45d425b853afcb6e"}
Jul 07 14:10:31.414712 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.414635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:31.783429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.783286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:31.900268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:31.900074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:32.083147 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.083060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:32.083309 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:32.083205    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:32.083309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.083291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:32.083458 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:32.083426    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:32.228516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.228479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5" event={"ID":"a779b2ca-332b-46f4-be32-69bddff5607b","Type":"ContainerStarted","Data":"0a81b0c997d2c807771cc8aa27d3a86640cbc0cd0c0db61d059a576e182b9395"}
Jul 07 14:10:32.597604 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.597571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:32.748619 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.748587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:32.780837 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.780803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:32.781462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.781436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:32.804258 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.804207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4tn5" podStartSLOduration=4.263275199 podStartE2EDuration="26.804189456s" totalImagesPullingTime="22.540914257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:32.258404607 +0000 UTC m=+26.814840970" watchObservedRunningTime="2026-07-07 14:10:32.804189456 +0000 UTC m=+27.360625823"
Jul 07 14:10:32.869593 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:32.869514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:33.082664 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:33.082625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:33.082847 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:33.082754    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:33.231814 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:33.231726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:33.232373 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:33.232147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bkxrc"
Jul 07 14:10:33.242883 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:33.242853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:33.351565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:33.351536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:34.084217 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:34.083405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:34.084217 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:34.083729    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:34.084217 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:34.083463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:34.084217 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:34.084198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:34.234868 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:34.234841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerStarted","Data":"010ac9a6f103d5f50c1e9652e5ff89c2a975a99d012fdac0d08706646d0bae89"}
Jul 07 14:10:34.238041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:34.238014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" event={"ID":"487a3a9c-267e-4ac8-82c5-00a1de9a0d9c","Type":"ContainerStarted","Data":"4953b9858fab438f59994dc7acac1481fdd266223b3b0fc5f356a069e09471c3"}
Jul 07 14:10:34.303550 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:34.303504    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7" podStartSLOduration=8.256075649 podStartE2EDuration="28.303488003s" totalImagesPullingTime="20.047412354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:34.30158675 +0000 UTC m=+28.858023115" watchObservedRunningTime="2026-07-07 14:10:34.303488003 +0000 UTC m=+28.859924367"
Jul 07 14:10:35.083047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.083020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:35.083275 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:35.083126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:35.243589 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.243552    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="010ac9a6f103d5f50c1e9652e5ff89c2a975a99d012fdac0d08706646d0bae89" exitCode=0
Jul 07 14:10:35.244056 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.243589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"010ac9a6f103d5f50c1e9652e5ff89c2a975a99d012fdac0d08706646d0bae89"}
Jul 07 14:10:35.244318 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.244302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:35.244412 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.244357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:35.244412 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.244389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:35.258892 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.258874    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:35.259078 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.259068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:10:35.770852 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.770801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:35.872568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.872536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:35.891725 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.891687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjsqs"]
Jul 07 14:10:35.891895 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.891831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:35.891955 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:35.891938    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:35.895309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.895282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hppq5"]
Jul 07 14:10:35.895446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.895388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:35.895503 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:35.895476    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:35.895973 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.895947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hp9vf"]
Jul 07 14:10:35.896148 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:35.896053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:35.896224 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:35.896162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:36.945959 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:36.945760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:37.053748 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.053710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:37.082609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.082585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:37.082701 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.082614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:37.082751 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:37.082694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:37.082831 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:37.082813    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:37.223320 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.223232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:37.249673 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.249650    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="553bfd5027be5ab5d67274687fe4a78ff40fc0593f5c2dfde0d3c41bfaac8c01" exitCode=0
Jul 07 14:10:37.249802 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.249727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"553bfd5027be5ab5d67274687fe4a78ff40fc0593f5c2dfde0d3c41bfaac8c01"}
Jul 07 14:10:37.303648 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.303613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:37.696627 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.696551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:37.793498 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:37.793466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:38.083230 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:38.083195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:38.083792 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:38.083357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:38.333736 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:38.333711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:38.476661 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:38.476633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h5x7b_cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b/dns-node-resolver/0.log"
Jul 07 14:10:38.493621 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:38.493594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:39.082784 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.082757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:39.082925 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.082794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:39.082925 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.082863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:39.082925 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.082917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:39.255821 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.255786    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="22ea3b38b608711068ad0f5317d413a33a4e15bacca685405dd6862fcfd9dccd" exitCode=0
Jul 07 14:10:39.256433 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.255845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"22ea3b38b608711068ad0f5317d413a33a4e15bacca685405dd6862fcfd9dccd"}
Jul 07 14:10:39.334877 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.334808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:39.626343 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.626308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:39.626508 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.626372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:39.626508 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.626483    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:39.626615 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.626548    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:39.626615 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.626552    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret podName:565c84fb-b78f-4739-bcd8-216275ac3643 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:11.626532311 +0000 UTC m=+66.182968656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret") pod "global-pull-secret-syncer-vjsqs" (UID: "565c84fb-b78f-4739-bcd8-216275ac3643") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:10:39.626615 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.626604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs podName:00f7d4fa-1d46-4d30-964e-ab7dac77df46 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:11.626591789 +0000 UTC m=+66.183028134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs") pod "network-metrics-daemon-hp9vf" (UID: "00f7d4fa-1d46-4d30-964e-ab7dac77df46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:10:39.680080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.680053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k8qgz_c85353ad-cb20-4ae6-8bf0-36406ff4c93e/node-ca/0.log"
Jul 07 14:10:39.727337 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:39.727303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:39.727514 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.727495    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:10:39.727584 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.727522    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:10:39.727584 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.727535    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5gpkt for pod openshift-network-diagnostics/network-check-target-hppq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:39.727677 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:39.727598    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt podName:44ed1af8-9484-4e0b-8546-436d65350450 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:11.727578927 +0000 UTC m=+66.284015271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gpkt" (UniqueName: "kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt") pod "network-check-target-hppq5" (UID: "44ed1af8-9484-4e0b-8546-436d65350450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:10:40.082717 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:40.082627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:40.082881 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:40.082754    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjsqs" podUID="565c84fb-b78f-4739-bcd8-216275ac3643"
Jul 07 14:10:40.712933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:40.712886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:40.958463 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:40.958433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:41.066862 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:41.066779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:41.082866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:41.082842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:41.082994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:41.082876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:41.082994 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:41.082954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hp9vf" podUID="00f7d4fa-1d46-4d30-964e-ab7dac77df46"
Jul 07 14:10:41.083128 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:10:41.083085    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hppq5" podUID="44ed1af8-9484-4e0b-8546-436d65350450"
Jul 07 14:10:41.260570 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:41.260541    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-100.ec2.internal" event="NodeReady"
Jul 07 14:10:41.260748 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:41.260686    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 14:10:42.083158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:42.083119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:10:42.086336 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:42.086311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:10:42.203587 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:42.203542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.083356 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.083322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:10:43.083777 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.083366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:10:43.088556 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.088534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:10:43.088818 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.088801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:10:43.089927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.089794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qhb9j\""
Jul 07 14:10:43.089927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.089805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:10:43.089927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.089857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k5zq\""
Jul 07 14:10:43.282085 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.282053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.425580 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.425548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.525213 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.525180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.626680 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.626651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.698161 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.698085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:43.982497 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:43.982278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:44.271043 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:44.270928    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="d53ceeb948bb559783913d89c7a62ba5bc5ac072828c859e9e8a2d5688bea42d" exitCode=0
Jul 07 14:10:44.271043 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:44.270974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"d53ceeb948bb559783913d89c7a62ba5bc5ac072828c859e9e8a2d5688bea42d"}
Jul 07 14:10:44.818388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:44.818354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:45.160838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.160810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:45.276871 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.276835    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cc543ad-56d9-474e-aac9-21e6f2e00224" containerID="0102e417fb6b279983f46553c04054a4f84d7a59efc0961ed0cc9ad8d7915be8" exitCode=0
Jul 07 14:10:45.277236 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.276886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerDied","Data":"0102e417fb6b279983f46553c04054a4f84d7a59efc0961ed0cc9ad8d7915be8"}
Jul 07 14:10:45.434155 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.434091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:45.556789 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.556751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:45.763310 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:45.763245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:46.173083 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.173040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:46.282005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.281972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" event={"ID":"6cc543ad-56d9-474e-aac9-21e6f2e00224","Type":"ContainerStarted","Data":"37eae3f35f61605590d92ca32f4a4e33706de6e9c772df0d7305f3ef2d7108ac"}
Jul 07 14:10:46.315525 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.315468    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xk4q9" podStartSLOduration=9.181985885 podStartE2EDuration="40.315449373s" totalImagesPullingTime="31.133463488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:46.313713895 +0000 UTC m=+40.870150270" watchObservedRunningTime="2026-07-07 14:10:46.315449373 +0000 UTC m=+40.871885738"
Jul 07 14:10:46.758022 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.757985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:46.858954 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.858927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:46.957716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:46.957682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:47.173899 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:47.173872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:47.539199 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:47.539121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:47.671478 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:47.671445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:47.985856 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:47.985826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:48.380912 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:48.380883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:48.475289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:48.475266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:48.576344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:48.576312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:48.985774 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:48.985746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:49.584877 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:49.584851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:50.241728 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:50.241697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:50.389149 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:50.389119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:50.581218 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:50.581150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:50.681377 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:50.681344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:50.973749 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:50.973718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:51.388220 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:51.388186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:51.970153 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:51.970081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:52.364888 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:52.364860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:52.590578 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:52.590533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:52.679286 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:52.679197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:52.962467 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:52.962369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:53.285145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:53.285048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:53.423120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:53.423067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:53.491342 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:53.491310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:53.777952 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:53.777924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:54.011141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:54.011085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:57.552590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:57.552560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:57.652636 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:57.652588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:57.746526 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:57.746494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:57.870668 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:57.870640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.038044 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.038007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.243795 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.243706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.323742 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.323715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.434652 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.434618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.517312 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.517231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.665566 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.665524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:58.746950 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:58.746922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:59.240041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:59.240012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:59.435425 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:59.435397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:59.538575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:10:59.538496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:00.067305 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:00.067263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:00.185192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:00.185161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:02.507194 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:02.507161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:02.686969 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:02.686941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:03.360772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.360741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"]
Jul 07 14:11:03.362867 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.362851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:03.367494 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.367471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8gnq5\""
Jul 07 14:11:03.368186 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.368168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 14:11:03.368296 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.368205    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 14:11:03.375306 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.375286    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k87n5"]
Jul 07 14:11:03.376875 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.376858    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-22hzd"]
Jul 07 14:11:03.377009 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.376992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.378273 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.378258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59ff86cfdf-clb4h"]
Jul 07 14:11:03.378379 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.378366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:03.379634 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.379618    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f4hzs"]
Jul 07 14:11:03.379781 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.379754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.381446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.381428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.384147 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 14:11:03.384275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 14:11:03.384380 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5cxmz\""
Jul 07 14:11:03.384380 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384145    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9nx\""
Jul 07 14:11:03.384380 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 14:11:03.384380 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384218    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 14:11:03.384576 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 14:11:03.384615 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384583    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 14:11:03.384775 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 14:11:03.384823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.384808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 14:11:03.385417 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.385404    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 14:11:03.385491 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.385475    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 14:11:03.385815 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.385798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mts99\""
Jul 07 14:11:03.385920 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.385815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 14:11:03.386126 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.386111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nlj6s\""
Jul 07 14:11:03.392814 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.392793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"]
Jul 07 14:11:03.394712 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.394695    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 14:11:03.395850 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.395827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24\") pod \"migrator-d57d58cc9-fscgf\" (UID: \"f4d69376-31a3-408b-be87-2f780c540cbd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:03.395934 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.395870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58f834c2-b8a4-4d60-8ee9-4b4294accac9-crio-socket\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.395934 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.395891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rzs\" (UniqueName: \"kubernetes.io/projected/5c9152bf-f945-4dab-9072-9fd61b940531-kube-api-access-v4rzs\") pod \"downloads-685bfb5f64-22hzd\" (UID: \"5c9152bf-f945-4dab-9072-9fd61b940531\") " pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:03.395934 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.395913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphq8\" (UniqueName: \"kubernetes.io/projected/dc389be8-7df3-4b36-abc0-68784e3a6f62-kube-api-access-kphq8\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.396074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.395955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58f834c2-b8a4-4d60-8ee9-4b4294accac9-data-volume\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.396074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc389be8-7df3-4b36-abc0-68784e3a6f62-metrics-tls\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.396189 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxns\" (UniqueName: \"kubernetes.io/projected/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-api-access-xtxns\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.396189 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc389be8-7df3-4b36-abc0-68784e3a6f62-config-volume\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.396189 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-tls\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a70b3d92-1a64-473c-8aff-a78188c1050b-ca-trust-extracted\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7ff\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-kube-api-access-zc7ff\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.396300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-certificates\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-installation-pull-secrets\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396477 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-bound-sa-token\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396477 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-image-registry-private-configuration\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396477 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc389be8-7df3-4b36-abc0-68784e3a6f62-tmp-dir\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.396477 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-trusted-ca\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.396477 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.396469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58f834c2-b8a4-4d60-8ee9-4b4294accac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.400170 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.400119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59ff86cfdf-clb4h"]
Jul 07 14:11:03.406728 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.406708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-22hzd"]
Jul 07 14:11:03.409740 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.409717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k87n5"]
Jul 07 14:11:03.410715 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.410687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f4hzs"]
Jul 07 14:11:03.470435 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.470406    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ss2mx"]
Jul 07 14:11:03.470592 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.470540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:03.472452 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.472438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.475173 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.475147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 14:11:03.475297 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.475266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nrfgr\""
Jul 07 14:11:03.475430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.475414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 14:11:03.475492 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.475457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 14:11:03.488845 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.488824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ss2mx"]
Jul 07 14:11:03.497294 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rzs\" (UniqueName: \"kubernetes.io/projected/5c9152bf-f945-4dab-9072-9fd61b940531-kube-api-access-v4rzs\") pod \"downloads-685bfb5f64-22hzd\" (UID: \"5c9152bf-f945-4dab-9072-9fd61b940531\") " pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:03.497406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphq8\" (UniqueName: \"kubernetes.io/projected/dc389be8-7df3-4b36-abc0-68784e3a6f62-kube-api-access-kphq8\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.497406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58f834c2-b8a4-4d60-8ee9-4b4294accac9-data-volume\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.497406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qk7b\" (UniqueName: \"kubernetes.io/projected/c1320315-4957-4380-bede-1ceaef9623d7-kube-api-access-5qk7b\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.497406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc389be8-7df3-4b36-abc0-68784e3a6f62-metrics-tls\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.497617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxns\" (UniqueName: \"kubernetes.io/projected/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-api-access-xtxns\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.497617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc389be8-7df3-4b36-abc0-68784e3a6f62-config-volume\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.497617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1320315-4957-4380-bede-1ceaef9623d7-cert\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.497746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-tls\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.497746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a70b3d92-1a64-473c-8aff-a78188c1050b-ca-trust-extracted\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.497746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497681    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58f834c2-b8a4-4d60-8ee9-4b4294accac9-data-volume\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.497746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7ff\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-kube-api-access-zc7ff\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.497746 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-certificates\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-installation-pull-secrets\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-bound-sa-token\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-image-registry-private-configuration\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc389be8-7df3-4b36-abc0-68784e3a6f62-tmp-dir\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-trusted-ca\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58f834c2-b8a4-4d60-8ee9-4b4294accac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24\") pod \"migrator-d57d58cc9-fscgf\" (UID: \"f4d69376-31a3-408b-be87-2f780c540cbd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:03.498002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.497992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58f834c2-b8a4-4d60-8ee9-4b4294accac9-crio-socket\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.498432 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.498091    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58f834c2-b8a4-4d60-8ee9-4b4294accac9-crio-socket\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.498432 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.498230    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc389be8-7df3-4b36-abc0-68784e3a6f62-tmp-dir\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.498432 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.498272    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc389be8-7df3-4b36-abc0-68784e3a6f62-config-volume\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.498432 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.498312    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a70b3d92-1a64-473c-8aff-a78188c1050b-ca-trust-extracted\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.499169 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.499073    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-trusted-ca\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.499389 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.499363    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.499637 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.499615    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-certificates\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.501909 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.501887    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58f834c2-b8a4-4d60-8ee9-4b4294accac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.501995 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.501909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-installation-pull-secrets\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.502035 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.502018    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc389be8-7df3-4b36-abc0-68784e3a6f62-metrics-tls\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.502078 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.502061    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-registry-tls\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.502176 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.502160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a70b3d92-1a64-473c-8aff-a78188c1050b-image-registry-private-configuration\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.506446 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:03.506425    2563 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found
Jul 07 14:11:03.506556 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:03.506450    2563 projected.go:196] Error preparing data for projected volume kube-api-access-nwv24 for pod openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf: configmap "openshift-service-ca.crt" not found
Jul 07 14:11:03.506556 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:03.506519    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24 podName:f4d69376-31a3-408b-be87-2f780c540cbd nodeName:}" failed. No retries permitted until 2026-07-07 14:11:04.006501536 +0000 UTC m=+58.562937887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nwv24" (UniqueName: "kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24") pod "migrator-d57d58cc9-fscgf" (UID: "f4d69376-31a3-408b-be87-2f780c540cbd") : configmap "openshift-service-ca.crt" not found
Jul 07 14:11:03.508290 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.508248    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rzs\" (UniqueName: \"kubernetes.io/projected/5c9152bf-f945-4dab-9072-9fd61b940531-kube-api-access-v4rzs\") pod \"downloads-685bfb5f64-22hzd\" (UID: \"5c9152bf-f945-4dab-9072-9fd61b940531\") " pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:03.508874 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.508839    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7ff\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-kube-api-access-zc7ff\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.508964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.508894    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphq8\" (UniqueName: \"kubernetes.io/projected/dc389be8-7df3-4b36-abc0-68784e3a6f62-kube-api-access-kphq8\") pod \"dns-default-k87n5\" (UID: \"dc389be8-7df3-4b36-abc0-68784e3a6f62\") " pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.509053 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.509036    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70b3d92-1a64-473c-8aff-a78188c1050b-bound-sa-token\") pod \"image-registry-59ff86cfdf-clb4h\" (UID: \"a70b3d92-1a64-473c-8aff-a78188c1050b\") " pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.509221 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.509201    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxns\" (UniqueName: \"kubernetes.io/projected/58f834c2-b8a4-4d60-8ee9-4b4294accac9-kube-api-access-xtxns\") pod \"insights-runtime-extractor-f4hzs\" (UID: \"58f834c2-b8a4-4d60-8ee9-4b4294accac9\") " pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.598397 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.598363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qk7b\" (UniqueName: \"kubernetes.io/projected/c1320315-4957-4380-bede-1ceaef9623d7-kube-api-access-5qk7b\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.598557 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.598409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1320315-4957-4380-bede-1ceaef9623d7-cert\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.600732 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.600716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1320315-4957-4380-bede-1ceaef9623d7-cert\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.608021 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.607999    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qk7b\" (UniqueName: \"kubernetes.io/projected/c1320315-4957-4380-bede-1ceaef9623d7-kube-api-access-5qk7b\") pod \"ingress-canary-ss2mx\" (UID: \"c1320315-4957-4380-bede-1ceaef9623d7\") " pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.686379 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.686294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:03.695006 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.694970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:03.702117 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.702082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:03.707718 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.707696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f4hzs"
Jul 07 14:11:03.737761 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.737649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:03.781019 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.780707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ss2mx"
Jul 07 14:11:03.907303 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.907273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k87n5"]
Jul 07 14:11:03.909520 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:03.909489    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc389be8_7df3_4b36_abc0_68784e3a6f62.slice/crio-1d0126308db73f9235f1f3d111a6651b041039069dfd7b2d19e889fde2a916fc WatchSource:0}: Error finding container 1d0126308db73f9235f1f3d111a6651b041039069dfd7b2d19e889fde2a916fc: Status 404 returned error can't find the container with id 1d0126308db73f9235f1f3d111a6651b041039069dfd7b2d19e889fde2a916fc
Jul 07 14:11:03.929084 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.929055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59ff86cfdf-clb4h"]
Jul 07 14:11:03.930170 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.930093    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-22hzd"]
Jul 07 14:11:03.931116 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:03.931072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70b3d92_1a64_473c_8aff_a78188c1050b.slice/crio-24538e72d30d7fa41b58c8286805e789057b1d1986b428a051122221f1a4c691 WatchSource:0}: Error finding container 24538e72d30d7fa41b58c8286805e789057b1d1986b428a051122221f1a4c691: Status 404 returned error can't find the container with id 24538e72d30d7fa41b58c8286805e789057b1d1986b428a051122221f1a4c691
Jul 07 14:11:03.932329 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:03.932305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9152bf_f945_4dab_9072_9fd61b940531.slice/crio-255f4d8cd03e93b4a18782949912f079195fa29332d3fcb6daf21ebe9ea6943c WatchSource:0}: Error finding container 255f4d8cd03e93b4a18782949912f079195fa29332d3fcb6daf21ebe9ea6943c: Status 404 returned error can't find the container with id 255f4d8cd03e93b4a18782949912f079195fa29332d3fcb6daf21ebe9ea6943c
Jul 07 14:11:03.943864 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.943795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f4hzs"]
Jul 07 14:11:03.946593 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:03.946571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f834c2_b8a4_4d60_8ee9_4b4294accac9.slice/crio-262d82a05052b9f6b72c72f3b13be48fa67613c0ed3429aa0946993f219fbe75 WatchSource:0}: Error finding container 262d82a05052b9f6b72c72f3b13be48fa67613c0ed3429aa0946993f219fbe75: Status 404 returned error can't find the container with id 262d82a05052b9f6b72c72f3b13be48fa67613c0ed3429aa0946993f219fbe75
Jul 07 14:11:03.966676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.966653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ss2mx"]
Jul 07 14:11:03.971979 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:03.971956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1320315_4957_4380_bede_1ceaef9623d7.slice/crio-74cb42f86f0f4be7e2361e09484edeaf7c5ceac3e9cc813def3c53c4ede773c7 WatchSource:0}: Error finding container 74cb42f86f0f4be7e2361e09484edeaf7c5ceac3e9cc813def3c53c4ede773c7: Status 404 returned error can't find the container with id 74cb42f86f0f4be7e2361e09484edeaf7c5ceac3e9cc813def3c53c4ede773c7
Jul 07 14:11:03.972952 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:03.972933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:04.101444 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.101415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24\") pod \"migrator-d57d58cc9-fscgf\" (UID: \"f4d69376-31a3-408b-be87-2f780c540cbd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:04.103614 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.103590    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/f4d69376-31a3-408b-be87-2f780c540cbd-kube-api-access-nwv24\") pod \"migrator-d57d58cc9-fscgf\" (UID: \"f4d69376-31a3-408b-be87-2f780c540cbd\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:04.271896 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.271819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"
Jul 07 14:11:04.279892 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.279804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:04.326057 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.326018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ss2mx" event={"ID":"c1320315-4957-4380-bede-1ceaef9623d7","Type":"ContainerStarted","Data":"74cb42f86f0f4be7e2361e09484edeaf7c5ceac3e9cc813def3c53c4ede773c7"}
Jul 07 14:11:04.327446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.327419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-22hzd" event={"ID":"5c9152bf-f945-4dab-9072-9fd61b940531","Type":"ContainerStarted","Data":"255f4d8cd03e93b4a18782949912f079195fa29332d3fcb6daf21ebe9ea6943c"}
Jul 07 14:11:04.328709 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.328686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4hzs" event={"ID":"58f834c2-b8a4-4d60-8ee9-4b4294accac9","Type":"ContainerStarted","Data":"262d82a05052b9f6b72c72f3b13be48fa67613c0ed3429aa0946993f219fbe75"}
Jul 07 14:11:04.330408 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.330377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k87n5" event={"ID":"dc389be8-7df3-4b36-abc0-68784e3a6f62","Type":"ContainerStarted","Data":"1d0126308db73f9235f1f3d111a6651b041039069dfd7b2d19e889fde2a916fc"}
Jul 07 14:11:04.332216 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.332192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h" event={"ID":"a70b3d92-1a64-473c-8aff-a78188c1050b","Type":"ContainerStarted","Data":"3a1b5fa6cbd89f2211394280386eeb705ebf976fe171e15e0d565c31ee3bfd5f"}
Jul 07 14:11:04.332300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.332230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h" event={"ID":"a70b3d92-1a64-473c-8aff-a78188c1050b","Type":"ContainerStarted","Data":"24538e72d30d7fa41b58c8286805e789057b1d1986b428a051122221f1a4c691"}
Jul 07 14:11:04.332437 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.332422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:04.369947 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.369893    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h" podStartSLOduration=1.36987349 podStartE2EDuration="1.36987349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:04.368670608 +0000 UTC m=+58.925106974" watchObservedRunningTime="2026-07-07 14:11:04.36987349 +0000 UTC m=+58.926309856"
Jul 07 14:11:04.412047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.411966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf"]
Jul 07 14:11:04.416164 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:04.416132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d69376_31a3_408b_be87_2f780c540cbd.slice/crio-78fdd6ee72604f3689c0bdcc67d34afe6dff5c5712ae5c8aaa70bdd13e699d9f WatchSource:0}: Error finding container 78fdd6ee72604f3689c0bdcc67d34afe6dff5c5712ae5c8aaa70bdd13e699d9f: Status 404 returned error can't find the container with id 78fdd6ee72604f3689c0bdcc67d34afe6dff5c5712ae5c8aaa70bdd13e699d9f
Jul 07 14:11:04.702743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.702634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:04.845750 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:04.845646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:05.097793 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:05.097743    2563 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0"
Jul 07 14:11:05.097990 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:05.097959    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container migrator start failed in pod migrator-d57d58cc9-fscgf_openshift-kube-storage-version-migrator(f4d69376-31a3-408b-be87-2f780c540cbd): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 07 14:11:05.099218 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:05.099167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"migrator\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"graceful-termination\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"]" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" podUID="f4d69376-31a3-408b-be87-2f780c540cbd"
Jul 07 14:11:05.337281 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.337243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" event={"ID":"f4d69376-31a3-408b-be87-2f780c540cbd","Type":"ContainerStarted","Data":"78fdd6ee72604f3689c0bdcc67d34afe6dff5c5712ae5c8aaa70bdd13e699d9f"}
Jul 07 14:11:05.340459 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:05.340396    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"migrator\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"graceful-termination\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"]" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" podUID="f4d69376-31a3-408b-be87-2f780c540cbd"
Jul 07 14:11:05.439272 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.439188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:05.667391 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.667322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:05.669550 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.669527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.672792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6rsp2\""
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.673056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.673414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.673614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.673906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 14:11:05.674823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.674126    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 14:11:05.683535 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.682596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:05.684250 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.684188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.713800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.713866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c4c\" (UniqueName: \"kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.713913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.713963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.713988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.714208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.714044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.797713 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.797614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:05.814981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.814948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815157 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c4c\" (UniqueName: \"kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815157 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815287 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815287 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815287 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.815893 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.815847    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.816171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.816123    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.816171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.816136    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.817941 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.817919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.818073 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.818050    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.825541 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.825518    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c4c\" (UniqueName: \"kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c\") pod \"console-6bd4855b8d-hlrhf\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") " pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:05.922125 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.922081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:05.989076 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.988834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6rsp2\""
Jul 07 14:11:05.997216 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:05.997192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:06.341795 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:06.341744    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"migrator\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"graceful-termination\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0bb1c60aa352f4838a3163d1d871b614ac713f17a23b7fbd3c43573c4a5cef0: reading manifest sha256:bb2b1c98d7b28b0961d3ebcafa1355ec98b06f213eb613556554319ac0d4c60d in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"]" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" podUID="f4d69376-31a3-408b-be87-2f780c540cbd"
Jul 07 14:11:07.267391 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.267355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dnd7"
Jul 07 14:11:07.559378 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.559302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:07.661192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.661164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:11:07.725224 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.725193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:11:07.725394 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.725305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fr95\" (UniqueName: \"kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.734881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.733528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.735908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.735870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:07.747130 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.747089    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 14:11:07.834070 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834070 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834321 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834321 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834321 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834480 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834480 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fr95\" (UniqueName: \"kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834848 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834816    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834912    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.834984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834933    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.835083 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.834982    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.836698 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.836676    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.836798 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.836782    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.850496 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.850472    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fr95\" (UniqueName: \"kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95\") pod \"console-84b9699788-zfkd7\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") " pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:07.880423 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:07.880399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:08.037439 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.037406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:08.198234 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.198198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:08.351236 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.351204    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"]
Jul 07 14:11:08.353339 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.353319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:08.356191 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.356160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bqbwk\""
Jul 07 14:11:08.356311 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.356222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 14:11:08.367172 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.367143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"]
Jul 07 14:11:08.439353 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.439313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:08.439689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.439669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4f94153-6931-4323-b818-3820cac7ea59-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-5r6p7\" (UID: \"d4f94153-6931-4323-b818-3820cac7ea59\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:08.540519 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.540439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4f94153-6931-4323-b818-3820cac7ea59-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-5r6p7\" (UID: \"d4f94153-6931-4323-b818-3820cac7ea59\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:08.543383 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.543359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4f94153-6931-4323-b818-3820cac7ea59-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-5r6p7\" (UID: \"d4f94153-6931-4323-b818-3820cac7ea59\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:08.664174 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.664140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:08.885686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:08.885655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:09.122354 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:09.122326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:09.460635 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:09.460610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:09.467258 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:09.467217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:09.705795 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:09.705762    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ecf37b_a560_4ffc_8d22_14c4c6631056.slice/crio-bd2af5f0ec89d27059f854c36bcb688189a656114466c1e9cc707e93aef8572b WatchSource:0}: Error finding container bd2af5f0ec89d27059f854c36bcb688189a656114466c1e9cc707e93aef8572b: Status 404 returned error can't find the container with id bd2af5f0ec89d27059f854c36bcb688189a656114466c1e9cc707e93aef8572b
Jul 07 14:11:09.881599 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:09.880964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:11:09.888999 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:09.887315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"]
Jul 07 14:11:09.892812 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:09.892693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f94153_6931_4323_b818_3820cac7ea59.slice/crio-9a763761bbfa528f518939de7284e6f654df9cb0af86163c9284e107faaf3115 WatchSource:0}: Error finding container 9a763761bbfa528f518939de7284e6f654df9cb0af86163c9284e107faaf3115: Status 404 returned error can't find the container with id 9a763761bbfa528f518939de7284e6f654df9cb0af86163c9284e107faaf3115
Jul 07 14:11:10.354094 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.354055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4hzs" event={"ID":"58f834c2-b8a4-4d60-8ee9-4b4294accac9","Type":"ContainerStarted","Data":"153951b5553b0e82f511e39b8235b2b586e5ca3ec2c2b34c51e694eccff2c31b"}
Jul 07 14:11:10.355843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.355816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k87n5" event={"ID":"dc389be8-7df3-4b36-abc0-68784e3a6f62","Type":"ContainerStarted","Data":"82021ee84365f4787814a9523eb00fb5246baae4a5d2b418af050bacbbb0005e"}
Jul 07 14:11:10.355981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.355867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k87n5" event={"ID":"dc389be8-7df3-4b36-abc0-68784e3a6f62","Type":"ContainerStarted","Data":"e46cffb972712b4fdf741859c7b56858e12a1d4067bf2217ee479b529a8d0829"}
Jul 07 14:11:10.355981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.355958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:10.357299 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.357272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7" event={"ID":"d4f94153-6931-4323-b818-3820cac7ea59","Type":"ContainerStarted","Data":"9a763761bbfa528f518939de7284e6f654df9cb0af86163c9284e107faaf3115"}
Jul 07 14:11:10.363047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.363011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ss2mx" event={"ID":"c1320315-4957-4380-bede-1ceaef9623d7","Type":"ContainerStarted","Data":"d8ffcc8bf1d1bfcc48e201f6a7e69aa6584c96b177623a548b5157c9d7691a00"}
Jul 07 14:11:10.364605 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.364581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4855b8d-hlrhf" event={"ID":"d7ecf37b-a560-4ffc-8d22-14c4c6631056","Type":"ContainerStarted","Data":"bd2af5f0ec89d27059f854c36bcb688189a656114466c1e9cc707e93aef8572b"}
Jul 07 14:11:10.366027 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.366008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9699788-zfkd7" event={"ID":"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada","Type":"ContainerStarted","Data":"0f0994091c6733b976081590e37cc4367741632f9112565e624a556aafe2bb51"}
Jul 07 14:11:10.380676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.380629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k87n5" podStartSLOduration=1.589950729 podStartE2EDuration="7.380613245s" totalImagesPullingTime="5.790662516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:10.380301467 +0000 UTC m=+64.936737843" watchObservedRunningTime="2026-07-07 14:11:10.380613245 +0000 UTC m=+64.937049610"
Jul 07 14:11:10.398972 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:10.398845    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ss2mx" podStartSLOduration=1.664713777 podStartE2EDuration="7.398826449s" totalImagesPullingTime="5.734112672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:10.397388592 +0000 UTC m=+64.953824957" watchObservedRunningTime="2026-07-07 14:11:10.398826449 +0000 UTC m=+64.955262813"
Jul 07 14:11:11.372010 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.371970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4hzs" event={"ID":"58f834c2-b8a4-4d60-8ee9-4b4294accac9","Type":"ContainerStarted","Data":"03e8402478610c83b994e0ab81fe3157562febfe5e4437606d3652f21bf05b1d"}
Jul 07 14:11:11.374693 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.373927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7" event={"ID":"d4f94153-6931-4323-b818-3820cac7ea59","Type":"ContainerStarted","Data":"c5dcaddec7b94485e920f7c1726de514952335ab4169939569122f0b55a4ac4f"}
Jul 07 14:11:11.374693 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.374435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:11.380823 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.380777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7"
Jul 07 14:11:11.418832 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.418555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-5r6p7" podStartSLOduration=2.2620385020000002 podStartE2EDuration="3.418535659s" totalImagesPullingTime="1.156497157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:11.395117689 +0000 UTC m=+65.951554051" watchObservedRunningTime="2026-07-07 14:11:11.418535659 +0000 UTC m=+65.974972024"
Jul 07 14:11:11.665030 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.664943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:11:11.665221 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.665154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:11:11.668657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.668269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:11:11.668657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.668513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:11:11.687125 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.686017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/565c84fb-b78f-4739-bcd8-216275ac3643-original-pull-secret\") pod \"global-pull-secret-syncer-vjsqs\" (UID: \"565c84fb-b78f-4739-bcd8-216275ac3643\") " pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:11:11.687125 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.687122    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f7d4fa-1d46-4d30-964e-ab7dac77df46-metrics-certs\") pod \"network-metrics-daemon-hp9vf\" (UID: \"00f7d4fa-1d46-4d30-964e-ab7dac77df46\") " pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:11:11.766176 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.765755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:11:11.769285 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.769062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:11:11.780140 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.780054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:11:11.793629 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.793604    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpkt\" (UniqueName: \"kubernetes.io/projected/44ed1af8-9484-4e0b-8546-436d65350450-kube-api-access-5gpkt\") pod \"network-check-target-hppq5\" (UID: \"44ed1af8-9484-4e0b-8546-436d65350450\") " pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:11:11.793770 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.793726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjsqs"
Jul 07 14:11:11.898874 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.898806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k5zq\""
Jul 07 14:11:11.906488 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.904375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qhb9j\""
Jul 07 14:11:11.906488 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.905694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hp9vf"
Jul 07 14:11:11.912363 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.912157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:11:11.975824 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:11.975728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjsqs"]
Jul 07 14:11:12.090600 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.090513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hppq5"]
Jul 07 14:11:12.107534 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.107423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hp9vf"]
Jul 07 14:11:12.112733 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:12.112564    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7d4fa_1d46_4d30_964e_ab7dac77df46.slice/crio-5c29179a2a2858e13362df123af40a09985d83dfe9f672f86feab090314d8319 WatchSource:0}: Error finding container 5c29179a2a2858e13362df123af40a09985d83dfe9f672f86feab090314d8319: Status 404 returned error can't find the container with id 5c29179a2a2858e13362df123af40a09985d83dfe9f672f86feab090314d8319
Jul 07 14:11:12.382930 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.382845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hppq5" event={"ID":"44ed1af8-9484-4e0b-8546-436d65350450","Type":"ContainerStarted","Data":"f55753d7f1b5bf0b548f3dcd9c8943d426cdd5ec15fc24934759d85f2348ff2b"}
Jul 07 14:11:12.385665 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.385420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:12.386163 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.386077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hp9vf" event={"ID":"00f7d4fa-1d46-4d30-964e-ab7dac77df46","Type":"ContainerStarted","Data":"5c29179a2a2858e13362df123af40a09985d83dfe9f672f86feab090314d8319"}
Jul 07 14:11:12.390049 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.390022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjsqs" event={"ID":"565c84fb-b78f-4739-bcd8-216275ac3643","Type":"ContainerStarted","Data":"60260074ea210e5eb5fd0c4c4ca51cfeceba60e74f48f3451c9affb500faf668"}
Jul 07 14:11:12.419644 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.418421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"]
Jul 07 14:11:12.422193 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.421437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.424774 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.424315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 14:11:12.424774 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.424550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 14:11:12.424927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.424826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l6fhg\""
Jul 07 14:11:12.425756 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.425734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 14:11:12.426166 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.425904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 14:11:12.426166 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.425939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 14:11:12.435292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.435186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"]
Jul 07 14:11:12.470779 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.470577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.470779 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.470653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7bz\" (UniqueName: \"kubernetes.io/projected/ae7cb200-27a5-4a05-9e4f-5879de435147-kube-api-access-rb7bz\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.470779 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.470685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7cb200-27a5-4a05-9e4f-5879de435147-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.470779 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.470736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.513905 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.513825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:12.573141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.571990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.573141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.572201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.573141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.572235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7bz\" (UniqueName: \"kubernetes.io/projected/ae7cb200-27a5-4a05-9e4f-5879de435147-kube-api-access-rb7bz\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.573141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.572266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7cb200-27a5-4a05-9e4f-5879de435147-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.573602 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.573545    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7cb200-27a5-4a05-9e4f-5879de435147-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.585392 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.585362    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.585659 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.585613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7cb200-27a5-4a05-9e4f-5879de435147-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.589241 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.589193    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7bz\" (UniqueName: \"kubernetes.io/projected/ae7cb200-27a5-4a05-9e4f-5879de435147-kube-api-access-rb7bz\") pod \"prometheus-operator-567f7b8b7c-ntdxf\" (UID: \"ae7cb200-27a5-4a05-9e4f-5879de435147\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:12.609902 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.609832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:12.735806 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:12.735720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"
Jul 07 14:11:14.096852 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:14.096811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:17.206206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.205943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:17.241505 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.241423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf"]
Jul 07 14:11:17.245935 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:17.245850    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7cb200_27a5_4a05_9e4f_5879de435147.slice/crio-2f2c50e9f1d58237e50ae21bc1c541d1a4034e89df3a6bee94f027ba4858790e WatchSource:0}: Error finding container 2f2c50e9f1d58237e50ae21bc1c541d1a4034e89df3a6bee94f027ba4858790e: Status 404 returned error can't find the container with id 2f2c50e9f1d58237e50ae21bc1c541d1a4034e89df3a6bee94f027ba4858790e
Jul 07 14:11:17.394064 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.393964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:17.411484 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.411433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hppq5" event={"ID":"44ed1af8-9484-4e0b-8546-436d65350450","Type":"ContainerStarted","Data":"70c3e892b0ef9618afe6de2c6a82e606342ded1d4a7b966d93a9fec4a316ade3"}
Jul 07 14:11:17.411696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.411679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:11:17.414400 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.414375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hp9vf" event={"ID":"00f7d4fa-1d46-4d30-964e-ab7dac77df46","Type":"ContainerStarted","Data":"3513cc0ea6cc9594401d9193098a58a46c7746404f9389f84a0d8a11cbed58a4"}
Jul 07 14:11:17.414505 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.414428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hp9vf" event={"ID":"00f7d4fa-1d46-4d30-964e-ab7dac77df46","Type":"ContainerStarted","Data":"597dff1a3488fdb70ef2f3983c1da7323a250d67e5110660da18193c7f197100"}
Jul 07 14:11:17.417046 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.417022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4855b8d-hlrhf" event={"ID":"d7ecf37b-a560-4ffc-8d22-14c4c6631056","Type":"ContainerStarted","Data":"04f12d9ebbab952e23636fcdd06c91857607a795c9e65c32cc5f40e14d421ebc"}
Jul 07 14:11:17.419238 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.419217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9699788-zfkd7" event={"ID":"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada","Type":"ContainerStarted","Data":"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"}
Jul 07 14:11:17.421080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.421054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf" event={"ID":"ae7cb200-27a5-4a05-9e4f-5879de435147","Type":"ContainerStarted","Data":"2f2c50e9f1d58237e50ae21bc1c541d1a4034e89df3a6bee94f027ba4858790e"}
Jul 07 14:11:17.423664 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.423400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4hzs" event={"ID":"58f834c2-b8a4-4d60-8ee9-4b4294accac9","Type":"ContainerStarted","Data":"0963f744d3028f809382ae3174e98cf3b5f401063b284fe4b7cfe03ea5c56b39"}
Jul 07 14:11:17.441622 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.441549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hppq5" podStartSLOduration=66.452458363 podStartE2EDuration="1m11.441529083s" totalImagesPullingTime="4.98907072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:17.440792644 +0000 UTC m=+71.997229009" watchObservedRunningTime="2026-07-07 14:11:17.441529083 +0000 UTC m=+71.997965428"
Jul 07 14:11:17.470165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.469964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b9699788-zfkd7" podStartSLOduration=3.283100274 podStartE2EDuration="10.4699285s" totalImagesPullingTime="7.186828226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:17.469017707 +0000 UTC m=+72.025454071" watchObservedRunningTime="2026-07-07 14:11:17.4699285 +0000 UTC m=+72.026364856"
Jul 07 14:11:17.499369 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.499317    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f4hzs" podStartSLOduration=1.565964905 podStartE2EDuration="14.499297023s" totalImagesPullingTime="12.933332118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:17.498762005 +0000 UTC m=+72.055198369" watchObservedRunningTime="2026-07-07 14:11:17.499297023 +0000 UTC m=+72.055733388"
Jul 07 14:11:17.534133 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.534003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bd4855b8d-hlrhf" podStartSLOduration=5.175922719 podStartE2EDuration="12.533987804s" totalImagesPullingTime="7.358065085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:17.533597237 +0000 UTC m=+72.090033601" watchObservedRunningTime="2026-07-07 14:11:17.533987804 +0000 UTC m=+72.090424171"
Jul 07 14:11:17.556369 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.556250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hp9vf" podStartSLOduration=66.602698584 podStartE2EDuration="1m11.556230905s" totalImagesPullingTime="4.953532321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:17.555708047 +0000 UTC m=+72.112144434" watchObservedRunningTime="2026-07-07 14:11:17.556230905 +0000 UTC m=+72.112667270"
Jul 07 14:11:17.760219 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.760182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:17.916490 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:17.916449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:18.038601 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:18.038497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:18.038601 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:18.038567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:18.045124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:18.045064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:18.432049 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:18.432016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:11:18.504269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:18.504239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:19.735013 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:19.734969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:19.923696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:19.923661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:20.376026 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:20.375996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k87n5"
Jul 07 14:11:21.180934 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:21.180898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:21.335807 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:21.335776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:22.557857 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:22.557822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:22.761221 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:22.761173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:23.302603 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:23.302569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:23.429229 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:23.429198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:23.681810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:23.681770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:23.781987 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:23.781952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:25.342279 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:25.342255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59ff86cfdf-clb4h"
Jul 07 14:11:25.547880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:25.547848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:25.676626 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:25.676584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:25.997911 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:25.997839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:27.954499 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:27.954411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:28.467395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.467356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjsqs" event={"ID":"565c84fb-b78f-4739-bcd8-216275ac3643","Type":"ContainerStarted","Data":"46b5dcfbc029cf7ab05d2e3eff07aa8c87ecae85518513408f4fd5ef4f5097a8"}
Jul 07 14:11:28.469524 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.469496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" event={"ID":"f4d69376-31a3-408b-be87-2f780c540cbd","Type":"ContainerStarted","Data":"9c465686a735919285773903a00317960d12c52cb2ced9ef8b1305c3872d40d5"}
Jul 07 14:11:28.469657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.469553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" event={"ID":"f4d69376-31a3-408b-be87-2f780c540cbd","Type":"ContainerStarted","Data":"c1e6ff04cb380301713562ec279814bb68a2575263a3de7704ce300c460380e8"}
Jul 07 14:11:28.471158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.471128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-22hzd" event={"ID":"5c9152bf-f945-4dab-9072-9fd61b940531","Type":"ContainerStarted","Data":"f09fb3f245a470ffe66959b7e4c6a604f346e26dcd9d5559aa46f7f71bbda95d"}
Jul 07 14:11:28.471284 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.471262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:28.473612 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.473585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf" event={"ID":"ae7cb200-27a5-4a05-9e4f-5879de435147","Type":"ContainerStarted","Data":"c5141f62f8f937b300f470f2338a42886c5f4ae3a24ac6314e86e5c1e3e093a3"}
Jul 07 14:11:28.473735 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.473631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf" event={"ID":"ae7cb200-27a5-4a05-9e4f-5879de435147","Type":"ContainerStarted","Data":"cfed3d5fff70cbe818c055574a6530ce7e2b8c72b168dd954eadc0f06eada059"}
Jul 07 14:11:28.486913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.486869    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vjsqs" podStartSLOduration=67.094123275 podStartE2EDuration="1m22.486855819s" totalImagesPullingTime="15.392732544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:28.485822879 +0000 UTC m=+83.042259253" watchObservedRunningTime="2026-07-07 14:11:28.486855819 +0000 UTC m=+83.043292183"
Jul 07 14:11:28.487883 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.487866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-22hzd"
Jul 07 14:11:28.506652 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.506601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-22hzd" podStartSLOduration=1.8147088249999999 podStartE2EDuration="25.506586015s" totalImagesPullingTime="23.69187719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:28.50516796 +0000 UTC m=+83.061604325" watchObservedRunningTime="2026-07-07 14:11:28.506586015 +0000 UTC m=+83.063022378"
Jul 07 14:11:28.549335 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.549276    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fscgf" podStartSLOduration=2.310424199 podStartE2EDuration="25.549254338s" totalImagesPullingTime="23.238830139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-07 14:11:28.548943483 +0000 UTC m=+83.105379849" watchObservedRunningTime="2026-07-07 14:11:28.549254338 +0000 UTC m=+83.105690703"
Jul 07 14:11:28.550336 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.550292    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ntdxf" podStartSLOduration=6.421598571 podStartE2EDuration="16.550276472s" totalImagesPullingTime="10.128677901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:28.531328563 +0000 UTC m=+83.087764928" watchObservedRunningTime="2026-07-07 14:11:28.550276472 +0000 UTC m=+83.106712838"
Jul 07 14:11:28.733653 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.733570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:28.862355 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:28.862317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:30.513553 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.513508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:30.695196 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.695151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:30.756719 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.756684    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"]
Jul 07 14:11:30.777924 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.777850    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"]
Jul 07 14:11:30.778123 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.777987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.782340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.782318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:11:30.782467 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.782318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jdwd2\""
Jul 07 14:11:30.782590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.782567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 14:11:30.790158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.790134    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"]
Jul 07 14:11:30.790291 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.790256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.790567 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.790548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"]
Jul 07 14:11:30.793762 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.793741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6qz74\""
Jul 07 14:11:30.794214 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.793973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 14:11:30.794214 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.794041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 14:11:30.794405 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.794379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:11:30.799537 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.799499    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w99jz"]
Jul 07 14:11:30.812294 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.812273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.815158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.815134    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 14:11:30.815262 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.815169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 14:11:30.815432 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.815413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6f247\""
Jul 07 14:11:30.815523 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.815487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 14:11:30.824955 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.824934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fss42\" (UniqueName: \"kubernetes.io/projected/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-kube-api-access-fss42\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.825058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.824991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.825058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.825016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.825180 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.825094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.833667 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.833648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:30.925765 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4f5d889-55eb-452a-919f-6fa70cbd4918-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.925765 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-metrics-client-ca\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-textfile\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.925974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fss42\" (UniqueName: \"kubernetes.io/projected/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-kube-api-access-fss42\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.926042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-wtmp\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-sys\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-root\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.926360 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjkj4\" (UniqueName: \"kubernetes.io/projected/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-api-access-sjkj4\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.926689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.926689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:30.926689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.926689 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.926461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr8f\" (UniqueName: \"kubernetes.io/projected/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-kube-api-access-rqr8f\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:30.927121 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.927078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.928715 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.928692    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.929016 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.928999    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.940920 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.940897    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fss42\" (UniqueName: \"kubernetes.io/projected/cd358bc3-d7d1-4c29-9c87-38fe1e10af7a-kube-api-access-fss42\") pod \"openshift-state-metrics-54cf96dfcf-mhrvm\" (UID: \"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:30.966816 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:30.966785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:31.027545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjkj4\" (UniqueName: \"kubernetes.io/projected/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-api-access-sjkj4\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr8f\" (UniqueName: \"kubernetes.io/projected/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-kube-api-access-rqr8f\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4f5d889-55eb-452a-919f-6fa70cbd4918-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-metrics-client-ca\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.027782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-textfile\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-wtmp\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-sys\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.027993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-root\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:31.028059    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 14:11:31.028212 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:11:31.028142    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls podName:2fd5d7c9-f3b7-4ead-8057-d666debb1c90 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:31.52812111 +0000 UTC m=+86.084557469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls") pod "node-exporter-w99jz" (UID: "2fd5d7c9-f3b7-4ead-8057-d666debb1c90") : secret "node-exporter-tls" not found
Jul 07 14:11:31.028700 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-metrics-client-ca\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028700 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4f5d889-55eb-452a-919f-6fa70cbd4918-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.028700 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028523    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-sys\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028700 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028645    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028700 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028660    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-wtmp\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.028948 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028903    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-textfile\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.029011 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.028946    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.029069 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.029020    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-root\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.029431 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.029386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.031395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.031353    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.031640 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.031621    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.031756 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.031733    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.036468 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.036449    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjkj4\" (UniqueName: \"kubernetes.io/projected/e4f5d889-55eb-452a-919f-6fa70cbd4918-kube-api-access-sjkj4\") pod \"kube-state-metrics-7c9dbb8b7d-jzd5g\" (UID: \"e4f5d889-55eb-452a-919f-6fa70cbd4918\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.036561 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.036519    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr8f\" (UniqueName: \"kubernetes.io/projected/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-kube-api-access-rqr8f\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.064117 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.064071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:31.089699 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.089668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"
Jul 07 14:11:31.101838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.101810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"
Jul 07 14:11:31.186456 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.186223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:31.275361 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.275267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm"]
Jul 07 14:11:31.278490 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:31.278433    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd358bc3_d7d1_4c29_9c87_38fe1e10af7a.slice/crio-0a733c364f91de02330e22e4c7147a74b0296498d118fdee9c30761201798277 WatchSource:0}: Error finding container 0a733c364f91de02330e22e4c7147a74b0296498d118fdee9c30761201798277: Status 404 returned error can't find the container with id 0a733c364f91de02330e22e4c7147a74b0296498d118fdee9c30761201798277
Jul 07 14:11:31.295853 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.295836    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g"]
Jul 07 14:11:31.298060 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:31.298036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5d889_55eb_452a_919f_6fa70cbd4918.slice/crio-dcb1b892c5f9da23d4533c29cd9577d4f79ffbea56e25546e8f94f3a14971125 WatchSource:0}: Error finding container dcb1b892c5f9da23d4533c29cd9577d4f79ffbea56e25546e8f94f3a14971125: Status 404 returned error can't find the container with id dcb1b892c5f9da23d4533c29cd9577d4f79ffbea56e25546e8f94f3a14971125
Jul 07 14:11:31.350815 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.350791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:31.492540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.492509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm" event={"ID":"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a","Type":"ContainerStarted","Data":"2701ff1046f88e4da14cf63566d88822404575d9c1387e8d9797bc9d2ab6bdc1"}
Jul 07 14:11:31.492665 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.492578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm" event={"ID":"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a","Type":"ContainerStarted","Data":"0a733c364f91de02330e22e4c7147a74b0296498d118fdee9c30761201798277"}
Jul 07 14:11:31.494023 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.493995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g" event={"ID":"e4f5d889-55eb-452a-919f-6fa70cbd4918","Type":"ContainerStarted","Data":"dcb1b892c5f9da23d4533c29cd9577d4f79ffbea56e25546e8f94f3a14971125"}
Jul 07 14:11:31.532662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.532597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.535364 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.535324    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fd5d7c9-f3b7-4ead-8057-d666debb1c90-node-exporter-tls\") pod \"node-exporter-w99jz\" (UID: \"2fd5d7c9-f3b7-4ead-8057-d666debb1c90\") " pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.537411 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.537390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:31.722300 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.722267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w99jz"
Jul 07 14:11:31.745611 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:31.745579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd5d7c9_f3b7_4ead_8057_d666debb1c90.slice/crio-bbf4e2aafb01c3b4bb7d21374cccd7dbdf266a82a470d6107e3a58a44115e4df WatchSource:0}: Error finding container bbf4e2aafb01c3b4bb7d21374cccd7dbdf266a82a470d6107e3a58a44115e4df: Status 404 returned error can't find the container with id bbf4e2aafb01c3b4bb7d21374cccd7dbdf266a82a470d6107e3a58a44115e4df
Jul 07 14:11:31.970353 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:31.970314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:32.103705 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.103675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:32.219531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.219489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:32.504939 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.504852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm" event={"ID":"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a","Type":"ContainerStarted","Data":"e0a23f642dcab505f1928ab9ab05e64ac06ecdfcb0ccc3ce0b88625a23979fea"}
Jul 07 14:11:32.507505 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.507447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99jz" event={"ID":"2fd5d7c9-f3b7-4ead-8057-d666debb1c90","Type":"ContainerStarted","Data":"bbf4e2aafb01c3b4bb7d21374cccd7dbdf266a82a470d6107e3a58a44115e4df"}
Jul 07 14:11:32.781113 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.780969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:32.883933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:32.883894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:33.786945 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:33.786908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:33.862615 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:33.862585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:33.967562 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:33.967497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:33.973491 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:33.973464    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85744b65d9-86mjq"]
Jul 07 14:11:34.033636 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.033602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85744b65d9-86mjq"]
Jul 07 14:11:34.033879 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.033752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.037563 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.037369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 14:11:34.037563 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.037368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 14:11:34.037563 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.037377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 14:11:34.038230 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.038209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-34bv3m28tje99\""
Jul 07 14:11:34.038390 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.038367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 14:11:34.038390 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.038378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 14:11:34.038737 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.038617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lwqz2\""
Jul 07 14:11:34.157556 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-grpc-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157895 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2373bf6b-da4b-43bd-a9d5-abebf9473971-metrics-client-ca\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157895 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2bk\" (UniqueName: \"kubernetes.io/projected/2373bf6b-da4b-43bd-a9d5-abebf9473971-kube-api-access-2h2bk\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.157895 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.157829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.221375 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.221341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:34.258504 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2373bf6b-da4b-43bd-a9d5-abebf9473971-metrics-client-ca\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2bk\" (UniqueName: \"kubernetes.io/projected/2373bf6b-da4b-43bd-a9d5-abebf9473971-kube-api-access-2h2bk\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258834 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258834 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258834 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.258989 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.258884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-grpc-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.259369 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.259345    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2373bf6b-da4b-43bd-a9d5-abebf9473971-metrics-client-ca\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.261865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.261839    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.261983 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.261867    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.261983 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.261913    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.262378 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.262338    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.262456 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.262432    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-thanos-querier-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.263058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.263031    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2373bf6b-da4b-43bd-a9d5-abebf9473971-secret-grpc-tls\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.268496 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.268476    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2bk\" (UniqueName: \"kubernetes.io/projected/2373bf6b-da4b-43bd-a9d5-abebf9473971-kube-api-access-2h2bk\") pod \"thanos-querier-85744b65d9-86mjq\" (UID: \"2373bf6b-da4b-43bd-a9d5-abebf9473971\") " pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.315684 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.315591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:34.347929 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.347900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:34.418225 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:34.418193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.189484 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.189225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85744b65d9-86mjq"]
Jul 07 14:11:35.191011 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:35.190985    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2373bf6b_da4b_43bd_a9d5_abebf9473971.slice/crio-eb0509c3cf93de4e03b6ec95038ac10f46ed3269172ca43b055bddc2a6155532 WatchSource:0}: Error finding container eb0509c3cf93de4e03b6ec95038ac10f46ed3269172ca43b055bddc2a6155532: Status 404 returned error can't find the container with id eb0509c3cf93de4e03b6ec95038ac10f46ed3269172ca43b055bddc2a6155532
Jul 07 14:11:35.232033 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.232007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.334430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.334351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.533237 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.532680    2563 generic.go:356] "Generic (PLEG): container finished" podID="2fd5d7c9-f3b7-4ead-8057-d666debb1c90" containerID="bfd04ae24ab55c337f750bae42850af7dd085d0e5d137a7ad850474505cae798" exitCode=0
Jul 07 14:11:35.533519 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.533383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99jz" event={"ID":"2fd5d7c9-f3b7-4ead-8057-d666debb1c90","Type":"ContainerDied","Data":"bfd04ae24ab55c337f750bae42850af7dd085d0e5d137a7ad850474505cae798"}
Jul 07 14:11:35.537706 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.537685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g" event={"ID":"e4f5d889-55eb-452a-919f-6fa70cbd4918","Type":"ContainerStarted","Data":"b8d368d8065a4b76013f8540f91ac5ad7f26be46c7d165ad565907a015d663de"}
Jul 07 14:11:35.538021 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.537927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g" event={"ID":"e4f5d889-55eb-452a-919f-6fa70cbd4918","Type":"ContainerStarted","Data":"df6b82f61e340852f1e517197b930715d36c9540b9ee677a25dacc7c8a7f4899"}
Jul 07 14:11:35.538021 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.537965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g" event={"ID":"e4f5d889-55eb-452a-919f-6fa70cbd4918","Type":"ContainerStarted","Data":"f506a85518262d715c7c73ed7d0886b2a8fb6d57e8cf5771bdef342a09745ff2"}
Jul 07 14:11:35.539732 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.539663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"eb0509c3cf93de4e03b6ec95038ac10f46ed3269172ca43b055bddc2a6155532"}
Jul 07 14:11:35.542256 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.542237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm" event={"ID":"cd358bc3-d7d1-4c29-9c87-38fe1e10af7a","Type":"ContainerStarted","Data":"e791a506b7f74a2aaaec083bcf2d99d39007b46a07475935e18920cbfb94aea2"}
Jul 07 14:11:35.570434 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.570402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.578071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.578047    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:11:35.618650 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.618625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:11:35.618793 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.618761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.624202 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.623415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mhrvm" podStartSLOduration=2.110505462 podStartE2EDuration="5.62338013s" totalImagesPullingTime="3.512874668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:35.621749512 +0000 UTC m=+90.178185903" watchObservedRunningTime="2026-07-07 14:11:35.62338013 +0000 UTC m=+90.179816494"
Jul 07 14:11:35.648174 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.648089    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jzd5g" podStartSLOduration=1.923493997 podStartE2EDuration="5.648071611s" totalImagesPullingTime="3.724577614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:35.647348124 +0000 UTC m=+90.203784540" watchObservedRunningTime="2026-07-07 14:11:35.648071611 +0000 UTC m=+90.204507976"
Jul 07 14:11:35.659633 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.659477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lph\" (UniqueName: \"kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.773865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.773787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.817825 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.817794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.874664 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.874838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.874838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.874838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lph\" (UniqueName: \"kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.874838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.875038 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.875038 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.874876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.875548 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.875405    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.875660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.875548    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.875881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.875828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.876324 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.876282    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.878643 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.878621    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.878643 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.878632    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.884481 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.884460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lph\" (UniqueName: \"kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph\") pod \"console-65586b58bd-xdv8s\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") " pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:35.918603 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.918557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:35.932965 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:35.932913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:36.141053 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.141020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:11:36.146706 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:36.146675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082cf26e_b533_4d7c_8c2c_96876f067702.slice/crio-0de3b90127aa5e106cc24a25a0fe717d5417e8ac1f59ffaf919ee6d9d5096a4e WatchSource:0}: Error finding container 0de3b90127aa5e106cc24a25a0fe717d5417e8ac1f59ffaf919ee6d9d5096a4e: Status 404 returned error can't find the container with id 0de3b90127aa5e106cc24a25a0fe717d5417e8ac1f59ffaf919ee6d9d5096a4e
Jul 07 14:11:36.547994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.547958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99jz" event={"ID":"2fd5d7c9-f3b7-4ead-8057-d666debb1c90","Type":"ContainerStarted","Data":"8dd0f45f87986ac6ccf0f058c745d43fb9e9329be299d384451b89c2e22d4ed0"}
Jul 07 14:11:36.548702 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.548021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99jz" event={"ID":"2fd5d7c9-f3b7-4ead-8057-d666debb1c90","Type":"ContainerStarted","Data":"380f09e07c09d075622c2b71fc381b73f3b42ddf744896b8fee4749e52553c0e"}
Jul 07 14:11:36.549567 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.549543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65586b58bd-xdv8s" event={"ID":"082cf26e-b533-4d7c-8c2c-96876f067702","Type":"ContainerStarted","Data":"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"}
Jul 07 14:11:36.549684 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.549585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65586b58bd-xdv8s" event={"ID":"082cf26e-b533-4d7c-8c2c-96876f067702","Type":"ContainerStarted","Data":"0de3b90127aa5e106cc24a25a0fe717d5417e8ac1f59ffaf919ee6d9d5096a4e"}
Jul 07 14:11:36.579139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.579073    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w99jz" podStartSLOduration=3.163147827 podStartE2EDuration="6.579057753s" totalImagesPullingTime="3.277069941s" totalInitContainerRuntime="138.839985ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:36.577253812 +0000 UTC m=+91.133690208" watchObservedRunningTime="2026-07-07 14:11:36.579057753 +0000 UTC m=+91.135494108"
Jul 07 14:11:36.607631 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:36.607585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65586b58bd-xdv8s" podStartSLOduration=1.6075707289999999 podStartE2EDuration="1.607570729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:36.606612719 +0000 UTC m=+91.163049074" watchObservedRunningTime="2026-07-07 14:11:36.607570729 +0000 UTC m=+91.164007090"
Jul 07 14:11:37.077765 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.077733    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:11:37.106819 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.106791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.107045 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.107015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:11:37.110148 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 14:11:37.110303 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110194    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 14:11:37.110303 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 14:11:37.110460 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110438    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 14:11:37.110575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2vghc\""
Jul 07 14:11:37.110804 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.110783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 14:11:37.111743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:37.111743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 14:11:37.111743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 14:11:37.111743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 14:11:37.111743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 14:11:37.112089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.111773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 14:11:37.112298 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.112276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 14:11:37.112610 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.112557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 14:11:37.113084 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.113066    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 14:11:37.113237 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.113135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3a7l046s3ir0\""
Jul 07 14:11:37.118980 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.118950    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 14:11:37.119820 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.119801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 14:11:37.186772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.186772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.186984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.186984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.186984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-web-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.186984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.186998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-config-out\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9f6\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-kube-api-access-kj9f6\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.187585 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.187334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.287888 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.287834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.287898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-config-out\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.287927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.287955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.287982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9f6\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-kube-api-access-kj9f6\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.288404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289898 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.289187    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.289898 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.289799    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.292507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292281    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.292507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.292507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292388    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.292507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-web-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.292507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292873    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.292966    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.293297    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.293392    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.293910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.294007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026918b7-21b5-47f0-b8b8-125ab13d6c77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.294128    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026918b7-21b5-47f0-b8b8-125ab13d6c77-config-out\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.294493 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.294471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.295293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.295255    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298000 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.297319    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298000 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.297665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298194 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.298168    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-web-config\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298194 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.298168    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298703 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.298659    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.298808 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.298786    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9f6\" (UniqueName: \"kubernetes.io/projected/026918b7-21b5-47f0-b8b8-125ab13d6c77-kube-api-access-kj9f6\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.300463 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.300444    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026918b7-21b5-47f0-b8b8-125ab13d6c77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026918b7-21b5-47f0-b8b8-125ab13d6c77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:37.421138 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:37.421078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:38.005778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.005757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:11:38.007326 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:11:38.007302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026918b7_21b5_47f0_b8b8_125ab13d6c77.slice/crio-0d85199e5db57682582d630da8016a61279b538db8a7e3805f3539d8d9392d76 WatchSource:0}: Error finding container 0d85199e5db57682582d630da8016a61279b538db8a7e3805f3539d8d9392d76: Status 404 returned error can't find the container with id 0d85199e5db57682582d630da8016a61279b538db8a7e3805f3539d8d9392d76
Jul 07 14:11:38.332236 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.332204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:38.482490 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.482467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:38.563189 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.563157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"45dd02cce860080787b18a2dcf208b09ca640205954c25ff2e577af52c922b7b"}
Jul 07 14:11:38.563349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.563249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"87279c82f563011fd41b1456edc904f2c5264bd117324abb74dfb1bc8d75d52e"}
Jul 07 14:11:38.563349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.563286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"2975c4ed5d998e3d99a39464ea3bc19865694edcba1f2087c68b0405251112ab"}
Jul 07 14:11:38.564320 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:38.564296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"0d85199e5db57682582d630da8016a61279b538db8a7e3805f3539d8d9392d76"}
Jul 07 14:11:40.576908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.576829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"d417fb9e4441e01c5caed08d98ff389339dd92ac050892c156003c8a3cd6602b"}
Jul 07 14:11:40.576908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.576902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"07d31b11be356cd53f09a5a5a04a20e58a6a44add77ceb7f28e75727a1b698ab"}
Jul 07 14:11:40.577406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.576938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" event={"ID":"2373bf6b-da4b-43bd-a9d5-abebf9473971","Type":"ContainerStarted","Data":"5c56e14a516a53f4efe11d045f10775bb069082cdfd66fa231d1226befade5e3"}
Jul 07 14:11:40.577406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.576981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:40.578395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.578372    2563 generic.go:356] "Generic (PLEG): container finished" podID="026918b7-21b5-47f0-b8b8-125ab13d6c77" containerID="c50c987c7c2ea424773602df81a6144d4c975f5d8d1e168d36f8b8e532793eec" exitCode=0
Jul 07 14:11:40.578493 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.578430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerDied","Data":"c50c987c7c2ea424773602df81a6144d4c975f5d8d1e168d36f8b8e532793eec"}
Jul 07 14:11:40.606839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:40.606801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq" podStartSLOduration=3.070889783 podStartE2EDuration="7.606788033s" totalImagesPullingTime="4.53589825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:40.605139766 +0000 UTC m=+95.161576150" watchObservedRunningTime="2026-07-07 14:11:40.606788033 +0000 UTC m=+95.163224397"
Jul 07 14:11:41.695526 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:41.695472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:11:41.821232 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:41.821200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:41.924375 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:41.924341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:44.459411 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.459338    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6bd4855b8d-hlrhf" podUID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" containerName="console" containerID="cri-o://04f12d9ebbab952e23636fcdd06c91857607a795c9e65c32cc5f40e14d421ebc" gracePeriod=15
Jul 07 14:11:44.598398 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.598374    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" containerID="04f12d9ebbab952e23636fcdd06c91857607a795c9e65c32cc5f40e14d421ebc" exitCode=0
Jul 07 14:11:44.598497 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.598452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4855b8d-hlrhf" event={"ID":"d7ecf37b-a560-4ffc-8d22-14c4c6631056","Type":"ContainerDied","Data":"04f12d9ebbab952e23636fcdd06c91857607a795c9e65c32cc5f40e14d421ebc"}
Jul 07 14:11:44.758034 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.757995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:44.862869 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.862839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863006 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.862899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863083 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863186 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863113    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5c4c\" (UniqueName: \"kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c\") pod \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\" (UID: \"d7ecf37b-a560-4ffc-8d22-14c4c6631056\") "
Jul 07 14:11:44.863245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863227    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config" (OuterVolumeSpecName: "console-config") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:11:44.863458 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863428    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:11:44.863544 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863491    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:44.863603 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.863566    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:11:44.865455 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.865421    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c" (OuterVolumeSpecName: "kube-api-access-x5c4c") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "kube-api-access-x5c4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:11:44.865557 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.865488    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:11:44.865557 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.865532    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7ecf37b-a560-4ffc-8d22-14c4c6631056" (UID: "d7ecf37b-a560-4ffc-8d22-14c4c6631056"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:11:44.964270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.964174    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-oauth-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:44.964270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.964201    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5c4c\" (UniqueName: \"kubernetes.io/projected/d7ecf37b-a560-4ffc-8d22-14c4c6631056-kube-api-access-x5c4c\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:44.964270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.964214    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:44.964270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.964227    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ecf37b-a560-4ffc-8d22-14c4c6631056-service-ca\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:44.964270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:44.964239    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ecf37b-a560-4ffc-8d22-14c4c6631056-console-oauth-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:11:45.603743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.603720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4855b8d-hlrhf"
Jul 07 14:11:45.604156 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.603756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4855b8d-hlrhf" event={"ID":"d7ecf37b-a560-4ffc-8d22-14c4c6631056","Type":"ContainerDied","Data":"bd2af5f0ec89d27059f854c36bcb688189a656114466c1e9cc707e93aef8572b"}
Jul 07 14:11:45.604156 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.603861    2563 scope.go:117] "RemoveContainer" containerID="04f12d9ebbab952e23636fcdd06c91857607a795c9e65c32cc5f40e14d421ebc"
Jul 07 14:11:45.607666 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.607637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"c97d811d07ed9bc69f45430a176c2e3e3ce99c0f89ebf645d881d63dbb8131fe"}
Jul 07 14:11:45.607835 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.607816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"6147b9cb58133be98dc276d73a6678707c7e00e28cd36ddd0fcd9413d96e9d6b"}
Jul 07 14:11:45.607959 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.607882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"0f470c39e0ccb6f989a68ceb0dec0ed998e77ecb3b498532483c4acc227547f5"}
Jul 07 14:11:45.607959 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.607937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"52c64ede30704d520c9fbf948c1821ea40bb358fde1619257a40aa647a9d07a5"}
Jul 07 14:11:45.608091 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.608065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"eda59c4290eab86818e6c21faf5ba97dfe67056e7a9c59531169afa250f523d4"}
Jul 07 14:11:45.608091 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.608119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026918b7-21b5-47f0-b8b8-125ab13d6c77","Type":"ContainerStarted","Data":"1bcd9140f5c29e0e55ab7e194bf253d9fb7ced7d0b2ccd126a57ea8eaaaa06e9"}
Jul 07 14:11:45.641331 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.641283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.682021013 podStartE2EDuration="8.641267203s" totalImagesPullingTime="5.942315255s" totalInitContainerRuntime="16.930935ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:45.639903519 +0000 UTC m=+100.196339899" watchObservedRunningTime="2026-07-07 14:11:45.641267203 +0000 UTC m=+100.197703567"
Jul 07 14:11:45.653243 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.653221    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:45.661092 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.661072    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bd4855b8d-hlrhf"]
Jul 07 14:11:45.933467 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:45.933438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:11:46.087740 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:46.087710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" path="/var/lib/kubelet/pods/d7ecf37b-a560-4ffc-8d22-14c4c6631056/volumes"
Jul 07 14:11:46.590382 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:46.590357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85744b65d9-86mjq"
Jul 07 14:11:47.421522 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:47.421491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:11:48.429609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:48.429581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hppq5"
Jul 07 14:11:52.999141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:52.999092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:11:53.139500 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:53.139468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:53.240446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:53.240418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:53.314275 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:53.314192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:11:53.381999 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:11:53.381966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:06.717250 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:06.717186    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65586b58bd-xdv8s" podUID="082cf26e-b533-4d7c-8c2c-96876f067702" containerName="console" containerID="cri-o://5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1" gracePeriod=15
Jul 07 14:12:06.995973 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:06.995951    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:12:07.053572 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053731 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053731 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053731 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5lph\" (UniqueName: \"kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053731 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.053913 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config\") pod \"082cf26e-b533-4d7c-8c2c-96876f067702\" (UID: \"082cf26e-b533-4d7c-8c2c-96876f067702\") "
Jul 07 14:12:07.054015 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.053982    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:07.054116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.054075    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-trusted-ca-bundle\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.054190 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.054125    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca" (OuterVolumeSpecName: "service-ca") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:07.054330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.054304    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config" (OuterVolumeSpecName: "console-config") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:07.054461 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.054338    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:07.055994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.055955    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:12:07.056092 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.056032    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:12:07.056092 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.056057    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph" (OuterVolumeSpecName: "kube-api-access-s5lph") pod "082cf26e-b533-4d7c-8c2c-96876f067702" (UID: "082cf26e-b533-4d7c-8c2c-96876f067702"). InnerVolumeSpecName "kube-api-access-s5lph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:12:07.155224 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155193    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-console-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.155224 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155220    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-oauth-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.155407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155233    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082cf26e-b533-4d7c-8c2c-96876f067702-service-ca\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.155407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155245    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5lph\" (UniqueName: \"kubernetes.io/projected/082cf26e-b533-4d7c-8c2c-96876f067702-kube-api-access-s5lph\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.155407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155257    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.155407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.155269    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082cf26e-b533-4d7c-8c2c-96876f067702-console-oauth-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:07.698413 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.698380    2563 generic.go:356] "Generic (PLEG): container finished" podID="082cf26e-b533-4d7c-8c2c-96876f067702" containerID="5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1" exitCode=0
Jul 07 14:12:07.698577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.698428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65586b58bd-xdv8s"
Jul 07 14:12:07.698577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.698431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65586b58bd-xdv8s" event={"ID":"082cf26e-b533-4d7c-8c2c-96876f067702","Type":"ContainerDied","Data":"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"}
Jul 07 14:12:07.698577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.698471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65586b58bd-xdv8s" event={"ID":"082cf26e-b533-4d7c-8c2c-96876f067702","Type":"ContainerDied","Data":"0de3b90127aa5e106cc24a25a0fe717d5417e8ac1f59ffaf919ee6d9d5096a4e"}
Jul 07 14:12:07.698577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.698490    2563 scope.go:117] "RemoveContainer" containerID="5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"
Jul 07 14:12:07.709692 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.709674    2563 scope.go:117] "RemoveContainer" containerID="5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"
Jul 07 14:12:07.709960 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:12:07.709942    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1\": container with ID starting with 5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1 not found: ID does not exist" containerID="5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"
Jul 07 14:12:07.710023 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.709970    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1"} err="failed to get container status \"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1\": rpc error: code = NotFound desc = could not find container \"5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1\": container with ID starting with 5f9abc33c1cd7dedd0e0f8b3bcf7a7aaa13f9458c8e9fbfedb033d4c53bc89a1 not found: ID does not exist"
Jul 07 14:12:07.722394 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.722375    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:12:07.728436 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:07.728417    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65586b58bd-xdv8s"]
Jul 07 14:12:08.087653 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:08.087575    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082cf26e-b533-4d7c-8c2c-96876f067702" path="/var/lib/kubelet/pods/082cf26e-b533-4d7c-8c2c-96876f067702/volumes"
Jul 07 14:12:10.505304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:10.505271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:10.619192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:10.619161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:18.019068 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.019003    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84b9699788-zfkd7" podUID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" containerName="console" containerID="cri-o://8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545" gracePeriod=15
Jul 07 14:12:18.300786 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.300763    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:12:18.347836 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.347808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.347982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.347880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.347982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.347930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fr95\" (UniqueName: \"kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.347982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.347961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.348173 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.347994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.348173 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.348173 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert\") pod \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\" (UID: \"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada\") "
Jul 07 14:12:18.348509 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348437    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:18.348509 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348444    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca" (OuterVolumeSpecName: "service-ca") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:18.348676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348450    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config" (OuterVolumeSpecName: "console-config") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:18.348676 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.348633    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:12:18.349936 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.349905    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:12:18.350407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.350386    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:12:18.350492 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.350438    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95" (OuterVolumeSpecName: "kube-api-access-4fr95") pod "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" (UID: "53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada"). InnerVolumeSpecName "kube-api-access-4fr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:12:18.449134 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449113    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-trusted-ca-bundle\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449134 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449132    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449141    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-oauth-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449149    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-oauth-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449159    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-console-serving-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449167    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fr95\" (UniqueName: \"kubernetes.io/projected/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-kube-api-access-4fr95\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.449254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.449176    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada-service-ca\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:12:18.743117 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.743071    2563 generic.go:356] "Generic (PLEG): container finished" podID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" containerID="8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545" exitCode=0
Jul 07 14:12:18.743269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.743133    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9699788-zfkd7"
Jul 07 14:12:18.743269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.743149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9699788-zfkd7" event={"ID":"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada","Type":"ContainerDied","Data":"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"}
Jul 07 14:12:18.743269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.743193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9699788-zfkd7" event={"ID":"53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada","Type":"ContainerDied","Data":"0f0994091c6733b976081590e37cc4367741632f9112565e624a556aafe2bb51"}
Jul 07 14:12:18.743269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.743212    2563 scope.go:117] "RemoveContainer" containerID="8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"
Jul 07 14:12:18.751677 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.751662    2563 scope.go:117] "RemoveContainer" containerID="8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"
Jul 07 14:12:18.751917 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:12:18.751899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545\": container with ID starting with 8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545 not found: ID does not exist" containerID="8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"
Jul 07 14:12:18.751966 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.751923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545"} err="failed to get container status \"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545\": rpc error: code = NotFound desc = could not find container \"8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545\": container with ID starting with 8834bc651ee63896a1ce3b633778e5959a12fdbd98d5923b81ee488c10a96545 not found: ID does not exist"
Jul 07 14:12:18.764491 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.764469    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:12:18.767657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:18.767639    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b9699788-zfkd7"]
Jul 07 14:12:20.087939 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:20.087907    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" path="/var/lib/kubelet/pods/53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada/volumes"
Jul 07 14:12:37.421576 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:37.421540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:12:37.440697 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:37.440671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:12:37.830991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:37.830916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:12:46.948156 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:46.948124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:47.008139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:47.008092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:57.644556 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:57.644486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:57.705900 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:57.705871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:57.935145 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:57.935047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:58.037107 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:58.037077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:58.752413 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:58.752385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:58.852994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:58.852967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:12:58.924092 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:12:58.924069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:03.308688 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:03.308655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:03.410082 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:03.410052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:08.295944 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:08.295906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:08.402148 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:08.402071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:08.496615 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:08.496581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:08.596756 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:08.596728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:21.923668 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:21.923624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:21.996997 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:21.996971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:33.399265 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:33.399234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:35.847894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:35.847864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:54.013805 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:54.013773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:54.079017 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:54.078985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:54.369082 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:54.369036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:54.469764 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:54.469734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:13:54.571933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:13:54.571910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:21.686263 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:21.686223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:21.781932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:21.781901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:26.620087 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:26.620056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:26.889033 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:26.888959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:27.116706 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:27.116651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:27.345208 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:27.345172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:27.618166 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:27.618049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:27.827629 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:27.827601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:27.985197 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:27.985089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:28.078706 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:28.078676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:28.245165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:28.245050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:28.471563 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:28.471533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:28.630568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:28.630539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:28.746504 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:28.746477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:41.899748 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:41.899706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:42.000121 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:42.000064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:43.680609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:43.680566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:51.817899 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:51.817865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:14:51.886216 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:14:51.886190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:15:05.876170 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:05.876142    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 14:15:35.972808 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:35.972777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:15:36.079425 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079393    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-scbn4"]
Jul 07 14:15:36.079768 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079750    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" containerName="console"
Jul 07 14:15:36.079855 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" containerName="console"
Jul 07 14:15:36.079855 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082cf26e-b533-4d7c-8c2c-96876f067702" containerName="console"
Jul 07 14:15:36.079855 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="082cf26e-b533-4d7c-8c2c-96876f067702" containerName="console"
Jul 07 14:15:36.079855 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079822    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" containerName="console"
Jul 07 14:15:36.079855 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079830    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" containerName="console"
Jul 07 14:15:36.080088 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079911    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ecf37b-a560-4ffc-8d22-14c4c6631056" containerName="console"
Jul 07 14:15:36.080088 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53cb4ae8-fcc6-41e8-9ef2-d6f2b9487ada" containerName="console"
Jul 07 14:15:36.080088 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.079937    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="082cf26e-b533-4d7c-8c2c-96876f067702" containerName="console"
Jul 07 14:15:36.081821 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.081795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.084682 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.084652    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d9968\""
Jul 07 14:15:36.084922 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.084900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 14:15:36.085008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.084911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 14:15:36.086008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.085991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 14:15:36.094560 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.094542    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-scbn4"]
Jul 07 14:15:36.202382 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.202353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2c9n\" (UniqueName: \"kubernetes.io/projected/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-kube-api-access-h2c9n\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.202523 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.202461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-data\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.303314 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.303242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-data\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.303314 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.303284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2c9n\" (UniqueName: \"kubernetes.io/projected/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-kube-api-access-h2c9n\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.303634 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.303612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-data\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.316371 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.316343    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2c9n\" (UniqueName: \"kubernetes.io/projected/ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec-kube-api-access-h2c9n\") pod \"seaweedfs-64568bcd49-scbn4\" (UID: \"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec\") " pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.391683 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.391654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:36.518016 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.517993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-scbn4"]
Jul 07 14:15:36.519930 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:15:36.519904    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce81d7c8_9e7b_4297_9257_0bfacf9fd5ec.slice/crio-0eeff95acc1dbdc362293395b8e279c93ef4c8beaf6504336225c68641c5310d WatchSource:0}: Error finding container 0eeff95acc1dbdc362293395b8e279c93ef4c8beaf6504336225c68641c5310d: Status 404 returned error can't find the container with id 0eeff95acc1dbdc362293395b8e279c93ef4c8beaf6504336225c68641c5310d
Jul 07 14:15:36.521087 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:36.521070    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:15:37.460954 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:37.460924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-scbn4" event={"ID":"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec","Type":"ContainerStarted","Data":"0eeff95acc1dbdc362293395b8e279c93ef4c8beaf6504336225c68641c5310d"}
Jul 07 14:15:40.010277 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:40.010235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:15:41.478938 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:41.478905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-scbn4" event={"ID":"ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec","Type":"ContainerStarted","Data":"9a575549d0000d2261dc07d181da670932584c3711c9983d9787d52e5033b66d"}
Jul 07 14:15:41.479396 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:41.478991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:15:41.498074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:41.498026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-scbn4" podStartSLOduration=1.5739524010000001 podStartE2EDuration="5.498012423s" totalImagesPullingTime="3.924060022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:15:41.497401864 +0000 UTC m=+336.053838228" watchObservedRunningTime="2026-07-07 14:15:41.498012423 +0000 UTC m=+336.054448787"
Jul 07 14:15:47.484801 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:15:47.484774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-scbn4"
Jul 07 14:16:19.604194 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:19.604084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:19.663643 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:19.663614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:45.217503 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:45.217470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:45.339675 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:45.339645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:45.430025 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:45.429994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:45.528224 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:45.528144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:46.239392 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:46.239359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:46.423015 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:46.422985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:16:50.738605 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:16:50.738570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:17:01.061657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.061625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-68nb5"]
Jul 07 14:17:01.064880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.064863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.076579 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.076541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-68nb5"]
Jul 07 14:17:01.081443 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.081411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdw6\" (UniqueName: \"kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.081619 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.081589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.182510 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.182474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.182699 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.182523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdw6\" (UniqueName: \"kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.182889 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.182868    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.193787 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.193752    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdw6\" (UniqueName: \"kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6\") pod \"s3-init-68nb5\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") " pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.326546 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.326469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:17:01.384147 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.384091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-68nb5"
Jul 07 14:17:01.430896 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.430276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:17:01.525270 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.525249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-68nb5"]
Jul 07 14:17:01.527804 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:17:01.527772    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod709c4649_a4c5_48bf_9d61_7a5f3851e937.slice/crio-e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403 WatchSource:0}: Error finding container e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403: Status 404 returned error can't find the container with id e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403
Jul 07 14:17:01.536338 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.536313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:17:01.774780 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:01.774749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerStarted","Data":"e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403"}
Jul 07 14:17:06.798041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:06.797957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerStarted","Data":"d14a95c7c51db6ecbb8eaf35883ee2d2759c45fbd62ef6bebce8bd9a2e5e33cf"}
Jul 07 14:17:08.808181 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:08.808151    2563 generic.go:356] "Generic (PLEG): container finished" podID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerID="d14a95c7c51db6ecbb8eaf35883ee2d2759c45fbd62ef6bebce8bd9a2e5e33cf" exitCode=0
Jul 07 14:17:08.808540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:08.808213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerDied","Data":"d14a95c7c51db6ecbb8eaf35883ee2d2759c45fbd62ef6bebce8bd9a2e5e33cf"}
Jul 07 14:17:13.832293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:13.832214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerStarted","Data":"9d2655f6cfd98578c50e6dbb6229d41ca267aabfc824c1f75f3bc473fec02180"}
Jul 07 14:17:13.858424 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:17:13.858231    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-68nb5" podStartSLOduration=1.462163034 podStartE2EDuration="12.85821234s" totalImagesPullingTime="9.588637189s" totalInitContainerRuntime="1.807412117s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:17:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:17:13.857239335 +0000 UTC m=+428.413675719" watchObservedRunningTime="2026-07-07 14:17:13.85821234 +0000 UTC m=+428.414648703"
Jul 07 14:18:16.366430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:16.366401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:18:19.086858 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:19.086779    2563 generic.go:356] "Generic (PLEG): container finished" podID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerID="9d2655f6cfd98578c50e6dbb6229d41ca267aabfc824c1f75f3bc473fec02180" exitCode=0
Jul 07 14:18:19.087231 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:19.086853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerDied","Data":"9d2655f6cfd98578c50e6dbb6229d41ca267aabfc824c1f75f3bc473fec02180"}
Jul 07 14:18:20.214713 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.214686    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-68nb5"
Jul 07 14:18:20.307230 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.307204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtdw6\" (UniqueName: \"kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6\") pod \"709c4649-a4c5-48bf-9d61-7a5f3851e937\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") "
Jul 07 14:18:20.307372 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.307238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache\") pod \"709c4649-a4c5-48bf-9d61-7a5f3851e937\" (UID: \"709c4649-a4c5-48bf-9d61-7a5f3851e937\") "
Jul 07 14:18:20.309350 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.309320    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6" (OuterVolumeSpecName: "kube-api-access-qtdw6") pod "709c4649-a4c5-48bf-9d61-7a5f3851e937" (UID: "709c4649-a4c5-48bf-9d61-7a5f3851e937"). InnerVolumeSpecName "kube-api-access-qtdw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:18:20.325596 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.325573    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache" (OuterVolumeSpecName: "model-cache") pod "709c4649-a4c5-48bf-9d61-7a5f3851e937" (UID: "709c4649-a4c5-48bf-9d61-7a5f3851e937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:18:20.407671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.407640    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709c4649-a4c5-48bf-9d61-7a5f3851e937-model-cache\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:18:20.407671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:20.407671    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtdw6\" (UniqueName: \"kubernetes.io/projected/709c4649-a4c5-48bf-9d61-7a5f3851e937-kube-api-access-qtdw6\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:18:21.095150 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:21.095094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-68nb5" event={"ID":"709c4649-a4c5-48bf-9d61-7a5f3851e937","Type":"ContainerDied","Data":"e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403"}
Jul 07 14:18:21.095150 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:21.095139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-68nb5"
Jul 07 14:18:21.095150 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:21.095148    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e154b7f140e94dea35e88e978fa28753010dae70c1d9e1ae836096cada35c403"
Jul 07 14:18:21.543865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:21.543791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:18:21.648267 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:21.648226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:18:33.497807 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.497771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:18:33.638737 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.638689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:18:33.697320 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:18:33.697638 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697627    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerName="s3-init"
Jul 07 14:18:33.697686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerName="s3-init"
Jul 07 14:18:33.697686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerName="download-hf-model"
Jul 07 14:18:33.697686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697668    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerName="download-hf-model"
Jul 07 14:18:33.697789 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.697726    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="709c4649-a4c5-48bf-9d61-7a5f3851e937" containerName="s3-init"
Jul 07 14:18:33.699793 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.699773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.702492 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.702471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8rhzg\""
Jul 07 14:18:33.703246 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.703226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-61b8a-predictor-serving-cert\""
Jul 07 14:18:33.703757 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.703743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:18:33.703857 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.703760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:18:33.703920 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.703898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:18:33.703920 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.703897    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-61b8a-kube-rbac-proxy-sar-config\""
Jul 07 14:18:33.704094 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.704078    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:18:33.714222 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.714202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:18:33.811136 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.811050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.811136 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.811082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mht\" (UniqueName: \"kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.811136 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.811128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.811330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.811157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.811330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.811178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912404 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mht\" (UniqueName: \"kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912540 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.912914 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.912894    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.913125 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.913084    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.913193 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.913122    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.914742 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.914724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:33.921440 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:33.921414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mht\" (UniqueName: \"kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht\") pod \"isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:34.010509 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:34.010472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:18:34.140812 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:34.140743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:18:34.143466 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:18:34.143439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc234c601_fef9_45f3_9668_ab81c7494147.slice/crio-9810ec3efd9c4470af1388ffccff2cd085e70545f87abb76400728e50e3a51b3 WatchSource:0}: Error finding container 9810ec3efd9c4470af1388ffccff2cd085e70545f87abb76400728e50e3a51b3: Status 404 returned error can't find the container with id 9810ec3efd9c4470af1388ffccff2cd085e70545f87abb76400728e50e3a51b3
Jul 07 14:18:35.148341 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:35.148311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerStarted","Data":"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"}
Jul 07 14:18:35.148724 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:35.148402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerStarted","Data":"9810ec3efd9c4470af1388ffccff2cd085e70545f87abb76400728e50e3a51b3"}
Jul 07 14:18:38.160538 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:38.160507    2563 generic.go:356] "Generic (PLEG): container finished" podID="c234c601-fef9-45f3-9668-ab81c7494147" containerID="352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1" exitCode=0
Jul 07 14:18:38.160899 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:38.160552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerDied","Data":"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"}
Jul 07 14:18:54.242708 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:54.242666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerStarted","Data":"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"}
Jul 07 14:18:57.257080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:18:57.257006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerStarted","Data":"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"}
Jul 07 14:19:00.275597 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:00.275562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerStarted","Data":"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"}
Jul 07 14:19:00.275992 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:00.275813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:19:00.275992 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:00.275871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:19:00.277279 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:00.277246    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:01.279606 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:01.279576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:19:01.280089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:01.279770    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:01.280841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:01.280813    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:01.283564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:01.283548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:19:01.316232 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:01.316184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podStartSLOduration=3.694955364 podStartE2EDuration="28.316166684s" totalImagesPullingTime="21.163208436s" totalInitContainerRuntime="3.458002884s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:18:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:19:00.311253655 +0000 UTC m=+534.867690019" watchObservedRunningTime="2026-07-07 14:19:01.316166684 +0000 UTC m=+535.872603048"
Jul 07 14:19:02.284649 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:02.284602    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:02.285063 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:02.285037    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:03.288982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:03.288939    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:03.289396 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:03.289350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:13.289048 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:13.289007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:13.289544 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:13.289488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:23.288890 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:23.288847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:23.289331 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:23.289305    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:33.289583 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:33.289533    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:33.289981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:33.289934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:43.289414 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:43.289374    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:43.289905 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:43.289883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:19:53.289021 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:53.288975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:19:53.289476 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:19:53.289437    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:20:03.288977 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:03.288926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:20:03.289427 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:03.289392    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:20:13.289267 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:13.289235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:13.289964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:13.289945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:28.835597 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.835432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:20:28.850201 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.850175    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:20:28.850474 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.850426    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" containerID="cri-o://1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e" gracePeriod=30
Jul 07 14:20:28.850474 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.850452    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" containerID="cri-o://86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664" gracePeriod=30
Jul 07 14:20:28.850620 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.850477    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" containerID="cri-o://e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7" gracePeriod=30
Jul 07 14:20:28.902005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.901981    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:20:28.905599 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.905581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:28.908732 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.908707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-884de-predictor-serving-cert\""
Jul 07 14:20:28.909178 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.909162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-884de-kube-rbac-proxy-sar-config\""
Jul 07 14:20:28.923079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.923053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:20:28.970093 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:28.970073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:20:29.000949 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.000926    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:20:29.004294 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.004278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.007086 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.007070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-884de-predictor-serving-cert\""
Jul 07 14:20:29.007350 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.007335    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-884de-kube-rbac-proxy-sar-config\""
Jul 07 14:20:29.017769 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.017750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:20:29.069742 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.069717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8x4k\" (UniqueName: \"kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.069910 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.069762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.069910 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.069781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.069910 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.069802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.069910 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.069872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.077596 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.077574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:20:29.170607 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8x4k\" (UniqueName: \"kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.170607 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.170800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.170800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.170800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.170800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.170986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.170986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.170986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4n2\" (UniqueName: \"kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.170986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.170968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.171390 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.171371    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.171603 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.171581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.171677 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.171581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.173304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.173285    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.182919 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.182900    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8x4k\" (UniqueName: \"kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k\") pod \"isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.218133 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.218115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:29.242626 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.242603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:20:29.271944 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.271916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4n2\" (UniqueName: \"kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.271974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.272032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.272058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.272095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272997 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.272947    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.272997 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.272987    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.273188 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.273080    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.274754 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.274734    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.282813 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.282784    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4n2\" (UniqueName: \"kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2\") pod \"isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.313680 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.313660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:29.354591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.354546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:20:29.361951 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.361929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:20:29.363369 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:20:29.363338    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2003c14_59bd_4497_9988_f52f04cfff1b.slice/crio-5805f11bbce610473cbbf014c789e99e80d1c029f3c1e145e0b2ab6605e69d0c WatchSource:0}: Error finding container 5805f11bbce610473cbbf014c789e99e80d1c029f3c1e145e0b2ab6605e69d0c: Status 404 returned error can't find the container with id 5805f11bbce610473cbbf014c789e99e80d1c029f3c1e145e0b2ab6605e69d0c
Jul 07 14:20:29.453822 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.453798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:20:29.454831 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:20:29.454812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03cc6cd_e0fc_4959_866a_b7d7d3d13cd1.slice/crio-524125b2d81f60f0134f0233e3db0c837f1b8e8a7644e68907edf806a0814965 WatchSource:0}: Error finding container 524125b2d81f60f0134f0233e3db0c837f1b8e8a7644e68907edf806a0814965: Status 404 returned error can't find the container with id 524125b2d81f60f0134f0233e3db0c837f1b8e8a7644e68907edf806a0814965
Jul 07 14:20:29.624806 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.624771    2563 generic.go:356] "Generic (PLEG): container finished" podID="c234c601-fef9-45f3-9668-ab81c7494147" containerID="e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7" exitCode=2
Jul 07 14:20:29.624983 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.624853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerDied","Data":"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"}
Jul 07 14:20:29.626377 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.626337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerStarted","Data":"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"}
Jul 07 14:20:29.626620 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.626406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerStarted","Data":"524125b2d81f60f0134f0233e3db0c837f1b8e8a7644e68907edf806a0814965"}
Jul 07 14:20:29.628075 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.628049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerStarted","Data":"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"}
Jul 07 14:20:29.628206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:29.628092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerStarted","Data":"5805f11bbce610473cbbf014c789e99e80d1c029f3c1e145e0b2ab6605e69d0c"}
Jul 07 14:20:31.280254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:31.280216    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:33.289679 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.289638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:20:33.290453 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.290433    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:20:33.648218 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.648184    2563 generic.go:356] "Generic (PLEG): container finished" podID="c234c601-fef9-45f3-9668-ab81c7494147" containerID="1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e" exitCode=0
Jul 07 14:20:33.648395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.648261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerDied","Data":"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"}
Jul 07 14:20:33.649565 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.649543    2563 generic.go:356] "Generic (PLEG): container finished" podID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerID="8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28" exitCode=0
Jul 07 14:20:33.649654 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.649609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerDied","Data":"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"}
Jul 07 14:20:33.651158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.651134    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerID="e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b" exitCode=0
Jul 07 14:20:33.651269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:33.651198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerDied","Data":"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"}
Jul 07 14:20:34.657970 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:34.657933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerStarted","Data":"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"}
Jul 07 14:20:34.658510 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:34.658009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerStarted","Data":"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"}
Jul 07 14:20:34.658510 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:34.658331    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:34.707719 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:34.707671    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podStartSLOduration=3.182544179 podStartE2EDuration="6.707653682s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.525109503s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:20:34.705232122 +0000 UTC m=+629.261668488" watchObservedRunningTime="2026-07-07 14:20:34.707653682 +0000 UTC m=+629.264090046"
Jul 07 14:20:35.663886 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:35.663857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:35.665572 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:35.665521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:20:36.280035 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:36.279989    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:36.669142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:36.669079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:20:41.280642 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:41.280592    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:41.281079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:41.280793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:41.673833 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:41.673801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:20:41.674560 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:41.674525    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:20:43.289090 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:43.289048    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:20:43.290653 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:43.290624    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:20:46.280597 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:46.280556    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:51.280325 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:51.280285    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:51.675215 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:51.675176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:20:53.289971 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:53.289918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 14:20:53.290429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:53.290140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:53.290769 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:53.290734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:20:53.290902 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:53.290861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:54.753481 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:54.753452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerStarted","Data":"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"}
Jul 07 14:20:54.753851 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:54.753510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerStarted","Data":"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"}
Jul 07 14:20:54.753851 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:54.753695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:54.783660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:54.783612    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podStartSLOduration=2.34261171 podStartE2EDuration="26.783597151s" totalImagesPullingTime="20.585447564s" totalInitContainerRuntime="3.855537877s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:20:54.782837322 +0000 UTC m=+649.339273698" watchObservedRunningTime="2026-07-07 14:20:54.783597151 +0000 UTC m=+649.340033515"
Jul 07 14:20:55.758454 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:55.758422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:20:55.759782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:55.759755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:20:56.279882 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:56.279841    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 14:20:56.762479 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:56.762441    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:20:59.003685 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.003663    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:59.142142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls\") pod \"c234c601-fef9-45f3-9668-ab81c7494147\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") "
Jul 07 14:20:59.142142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert\") pod \"c234c601-fef9-45f3-9668-ab81c7494147\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") "
Jul 07 14:20:59.142142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config\") pod \"c234c601-fef9-45f3-9668-ab81c7494147\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") "
Jul 07 14:20:59.142424 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location\") pod \"c234c601-fef9-45f3-9668-ab81c7494147\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") "
Jul 07 14:20:59.142424 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142222    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8mht\" (UniqueName: \"kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht\") pod \"c234c601-fef9-45f3-9668-ab81c7494147\" (UID: \"c234c601-fef9-45f3-9668-ab81c7494147\") "
Jul 07 14:20:59.142573 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142543    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c234c601-fef9-45f3-9668-ab81c7494147" (UID: "c234c601-fef9-45f3-9668-ab81c7494147"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:20:59.142573 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142556    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c234c601-fef9-45f3-9668-ab81c7494147" (UID: "c234c601-fef9-45f3-9668-ab81c7494147"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:20:59.142648 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.142553    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c234c601-fef9-45f3-9668-ab81c7494147" (UID: "c234c601-fef9-45f3-9668-ab81c7494147"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:20:59.144446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.144418    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht" (OuterVolumeSpecName: "kube-api-access-d8mht") pod "c234c601-fef9-45f3-9668-ab81c7494147" (UID: "c234c601-fef9-45f3-9668-ab81c7494147"). InnerVolumeSpecName "kube-api-access-d8mht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:20:59.144446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.144431    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c234c601-fef9-45f3-9668-ab81c7494147" (UID: "c234c601-fef9-45f3-9668-ab81c7494147"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:20:59.243631 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.243601    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mht\" (UniqueName: \"kubernetes.io/projected/c234c601-fef9-45f3-9668-ab81c7494147-kube-api-access-d8mht\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:20:59.243631 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.243631    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c234c601-fef9-45f3-9668-ab81c7494147-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:20:59.243821 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.243644    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:20:59.243821 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.243657    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c234c601-fef9-45f3-9668-ab81c7494147-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:20:59.243821 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.243668    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c234c601-fef9-45f3-9668-ab81c7494147-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:20:59.777074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.777040    2563 generic.go:356] "Generic (PLEG): container finished" podID="c234c601-fef9-45f3-9668-ab81c7494147" containerID="86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664" exitCode=0
Jul 07 14:20:59.777247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.777115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerDied","Data":"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"}
Jul 07 14:20:59.777247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.777122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"
Jul 07 14:20:59.777247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.777170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst" event={"ID":"c234c601-fef9-45f3-9668-ab81c7494147","Type":"ContainerDied","Data":"9810ec3efd9c4470af1388ffccff2cd085e70545f87abb76400728e50e3a51b3"}
Jul 07 14:20:59.777247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.777199    2563 scope.go:117] "RemoveContainer" containerID="86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"
Jul 07 14:20:59.785499 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.785482    2563 scope.go:117] "RemoveContainer" containerID="e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"
Jul 07 14:20:59.792661 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.792645    2563 scope.go:117] "RemoveContainer" containerID="1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"
Jul 07 14:20:59.799524 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.799506    2563 scope.go:117] "RemoveContainer" containerID="352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"
Jul 07 14:20:59.806593 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.806576    2563 scope.go:117] "RemoveContainer" containerID="86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"
Jul 07 14:20:59.806853 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:20:59.806835    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664\": container with ID starting with 86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664 not found: ID does not exist" containerID="86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"
Jul 07 14:20:59.806968 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.806858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664"} err="failed to get container status \"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664\": rpc error: code = NotFound desc = could not find container \"86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664\": container with ID starting with 86f757f4fb3c630d8fb3dee7c9a7ca09f883a0c095ad6626a0f2a4f2a3699664 not found: ID does not exist"
Jul 07 14:20:59.806968 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.806887    2563 scope.go:117] "RemoveContainer" containerID="e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"
Jul 07 14:20:59.807220 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:20:59.807126    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7\": container with ID starting with e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7 not found: ID does not exist" containerID="e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"
Jul 07 14:20:59.807220 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.807156    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7"} err="failed to get container status \"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7\": rpc error: code = NotFound desc = could not find container \"e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7\": container with ID starting with e187e18b0dcef29a0ebe533de0544a81082a7a8d87fce7df9af6fed38feee4e7 not found: ID does not exist"
Jul 07 14:20:59.807220 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.807176    2563 scope.go:117] "RemoveContainer" containerID="1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"
Jul 07 14:20:59.807492 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:20:59.807437    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e\": container with ID starting with 1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e not found: ID does not exist" containerID="1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"
Jul 07 14:20:59.807545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.807498    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e"} err="failed to get container status \"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e\": rpc error: code = NotFound desc = could not find container \"1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e\": container with ID starting with 1a05523835dc6ec0b4449942ae7d807ba75e7e11afbfe58dab0fa0e1aeaf3b2e not found: ID does not exist"
Jul 07 14:20:59.807545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.807514    2563 scope.go:117] "RemoveContainer" containerID="352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"
Jul 07 14:20:59.807726 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:20:59.807709    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1\": container with ID starting with 352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1 not found: ID does not exist" containerID="352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"
Jul 07 14:20:59.807802 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.807733    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1"} err="failed to get container status \"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1\": rpc error: code = NotFound desc = could not find container \"352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1\": container with ID starting with 352106cb8564429b3fe15a65a36fe1b1ba43f5e4ab631dafd2df62f513c692e1 not found: ID does not exist"
Jul 07 14:20:59.808539 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.808522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:20:59.817171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:20:59.817154    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-61b8a-predictor-745cfc7d54-d6gst"]
Jul 07 14:21:00.088736 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:00.088647    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c234c601-fef9-45f3-9668-ab81c7494147" path="/var/lib/kubelet/pods/c234c601-fef9-45f3-9668-ab81c7494147/volumes"
Jul 07 14:21:01.675394 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:01.675352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:21:01.768299 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:01.768269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:21:01.768900 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:01.768878    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:21:11.674880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:11.674835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:21:11.769782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:11.769745    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:21:21.674968 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:21.674925    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:21:21.769187 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:21.769138    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:21:31.674414 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:31.674376    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:21:31.768969 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:31.768927    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:21:32.738364 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:32.738329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:21:41.192982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:41.192947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:21:41.675113 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:41.675057    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:21:41.769052 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:41.769011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:21:51.675480 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:51.675446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:21:51.768935 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:21:51.768896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:22:01.770318 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:01.770235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:22:08.879713 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:08.879682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:09.672838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:09.672806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:18.931996 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:18.931967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:19.185733 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.185652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:19.248007 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.247976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:22:19.248268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.248228    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" containerID="cri-o://5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7" gracePeriod=30
Jul 07 14:22:19.248332 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.248266    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kube-rbac-proxy" containerID="cri-o://bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de" gracePeriod=30
Jul 07 14:22:19.250991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.250968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:22:19.251417 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container"
Jul 07 14:22:19.251417 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251446    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251466    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251474    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="storage-initializer"
Jul 07 14:22:19.251568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251496    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="storage-initializer"
Jul 07 14:22:19.251875 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251570    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="agent"
Jul 07 14:22:19.251875 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251584    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kserve-container"
Jul 07 14:22:19.251875 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.251598    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c234c601-fef9-45f3-9668-ab81c7494147" containerName="kube-rbac-proxy"
Jul 07 14:22:19.254089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.254072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.257813 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.257437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bd33a-kube-rbac-proxy-sar-config\""
Jul 07 14:22:19.257813 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.257667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bd33a-predictor-serving-cert\""
Jul 07 14:22:19.273141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.273122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:22:19.316079 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.316046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.316227 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.316086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.316227 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.316163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.316318 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.316234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.316357 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.316322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg8v\" (UniqueName: \"kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.325885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.325856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:19.416645 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.416621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg8v\" (UniqueName: \"kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.416772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.416666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.416772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.416689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.416772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.416712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.416772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.416741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.417173 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.417150    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.417307 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.417288    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.417407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.417387    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.418932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.418916    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.429405 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.429381    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg8v\" (UniqueName: \"kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v\") pod \"isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.452768 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.452709    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:22:19.456535 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.456505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.460612 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.460593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-bd33a-predictor-serving-cert\""
Jul 07 14:22:19.462111 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.462080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-bd33a-kube-rbac-proxy-sar-config\""
Jul 07 14:22:19.517723 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.517699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq6pt\" (UniqueName: \"kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.517861 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.517767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.517861 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.517784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.517861 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.517801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.517861 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.517816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.519272 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.519246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:19.546623 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.546595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:22:19.560410 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.560390    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:22:19.560621 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.560599    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" containerID="cri-o://b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65" gracePeriod=30
Jul 07 14:22:19.560683 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.560637    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kube-rbac-proxy" containerID="cri-o://0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe" gracePeriod=30
Jul 07 14:22:19.570281 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.570259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:19.620171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.619301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.619346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.619383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.619414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620171 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.619472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq6pt\" (UniqueName: \"kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620604 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.620583    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620941 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.620919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.620988 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.620958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.622322 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.622300    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.630218 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.630199    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq6pt\" (UniqueName: \"kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt\") pod \"isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.635366 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.635341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:22:19.701329 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.701305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:22:19.703127 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:22:19.703052    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe340e8_5229_4970_b432_41feb1f12ed2.slice/crio-46fccb341f8df27158d3b73a9f205bce1ea40065cc81cd12c59c4a3f518d2d0e WatchSource:0}: Error finding container 46fccb341f8df27158d3b73a9f205bce1ea40065cc81cd12c59c4a3f518d2d0e: Status 404 returned error can't find the container with id 46fccb341f8df27158d3b73a9f205bce1ea40065cc81cd12c59c4a3f518d2d0e
Jul 07 14:22:19.770594 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.770551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:19.911335 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:19.911316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:22:19.912795 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:22:19.912771    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95a58c6_8d8c_4249_80db_35ef7ca89c2d.slice/crio-42bd767f382ba6894bc118ac1d8a3faf6c01bafb201ab0e359c635206dcf83c8 WatchSource:0}: Error finding container 42bd767f382ba6894bc118ac1d8a3faf6c01bafb201ab0e359c635206dcf83c8: Status 404 returned error can't find the container with id 42bd767f382ba6894bc118ac1d8a3faf6c01bafb201ab0e359c635206dcf83c8
Jul 07 14:22:20.112817 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.112781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerStarted","Data":"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"}
Jul 07 14:22:20.113315 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.112850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerStarted","Data":"42bd767f382ba6894bc118ac1d8a3faf6c01bafb201ab0e359c635206dcf83c8"}
Jul 07 14:22:20.114615 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.114570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerStarted","Data":"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"}
Jul 07 14:22:20.114764 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.114638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerStarted","Data":"46fccb341f8df27158d3b73a9f205bce1ea40065cc81cd12c59c4a3f518d2d0e"}
Jul 07 14:22:20.116714 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.116691    2563 generic.go:356] "Generic (PLEG): container finished" podID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerID="0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe" exitCode=2
Jul 07 14:22:20.116840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.116766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerDied","Data":"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"}
Jul 07 14:22:20.118908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.118885    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerID="bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de" exitCode=2
Jul 07 14:22:20.118908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:20.118903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerDied","Data":"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"}
Jul 07 14:22:21.669884 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:21.669847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 14:22:21.675313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:21.675285    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:22:21.762664 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:21.762630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 14:22:21.769155 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:21.769128    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:22:23.105869 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.105849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:22:23.131697 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.131642    2563 generic.go:356] "Generic (PLEG): container finished" podID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerID="b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65" exitCode=0
Jul 07 14:22:23.131697 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.131690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"
Jul 07 14:22:23.131843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.131717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerDied","Data":"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"}
Jul 07 14:22:23.131843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.131777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc" event={"ID":"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1","Type":"ContainerDied","Data":"524125b2d81f60f0134f0233e3db0c837f1b8e8a7644e68907edf806a0814965"}
Jul 07 14:22:23.131843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.131803    2563 scope.go:117] "RemoveContainer" containerID="0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"
Jul 07 14:22:23.140314 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.140301    2563 scope.go:117] "RemoveContainer" containerID="b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"
Jul 07 14:22:23.147465 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147445    2563 scope.go:117] "RemoveContainer" containerID="8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"
Jul 07 14:22:23.147680 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config\") pod \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") "
Jul 07 14:22:23.147749 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn4n2\" (UniqueName: \"kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2\") pod \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") "
Jul 07 14:22:23.147810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert\") pod \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") "
Jul 07 14:22:23.147810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls\") pod \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") "
Jul 07 14:22:23.147918 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.147846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location\") pod \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\" (UID: \"a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1\") "
Jul 07 14:22:23.148139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.148113    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" (UID: "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:22:23.148219 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.148192    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:23.148467 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.148435    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" (UID: "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:22:23.148578 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.148536    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" (UID: "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:22:23.150185 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.150158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" (UID: "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:22:23.150316 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.150299    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2" (OuterVolumeSpecName: "kube-api-access-vn4n2") pod "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" (UID: "a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1"). InnerVolumeSpecName "kube-api-access-vn4n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:22:23.163536 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.163520    2563 scope.go:117] "RemoveContainer" containerID="0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"
Jul 07 14:22:23.163788 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:23.163769    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe\": container with ID starting with 0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe not found: ID does not exist" containerID="0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"
Jul 07 14:22:23.163841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.163797    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe"} err="failed to get container status \"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe\": rpc error: code = NotFound desc = could not find container \"0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe\": container with ID starting with 0a4221882a52bf1cd3e8d422b914968e28fd08845cdf49ced803091f768d59fe not found: ID does not exist"
Jul 07 14:22:23.163841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.163816    2563 scope.go:117] "RemoveContainer" containerID="b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"
Jul 07 14:22:23.164044 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:23.164028    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65\": container with ID starting with b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65 not found: ID does not exist" containerID="b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"
Jul 07 14:22:23.164089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.164051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65"} err="failed to get container status \"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65\": rpc error: code = NotFound desc = could not find container \"b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65\": container with ID starting with b5cc0bb2671bbfd23a82766983b3d60a156a50e3a10ba8b828c1a8bb1e3b9d65 not found: ID does not exist"
Jul 07 14:22:23.164089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.164070    2563 scope.go:117] "RemoveContainer" containerID="8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"
Jul 07 14:22:23.164305 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:23.164290    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28\": container with ID starting with 8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28 not found: ID does not exist" containerID="8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"
Jul 07 14:22:23.164345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.164308    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28"} err="failed to get container status \"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28\": rpc error: code = NotFound desc = could not find container \"8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28\": container with ID starting with 8e97d22d1e0baf5cd2d4f03f484b9000f41571e8aed438a7289d9dd2f7b27b28 not found: ID does not exist"
Jul 07 14:22:23.248894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.248862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4n2\" (UniqueName: \"kubernetes.io/projected/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kube-api-access-vn4n2\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:23.248894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.248894    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:23.248894 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.248904    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:23.249075 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.248913    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:23.457437 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.457412    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:22:23.464546 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.464520    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-884de-predictor-867f8cd589-k9mcc"]
Jul 07 14:22:23.984057 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:23.984034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:22:24.054810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.054728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8x4k\" (UniqueName: \"kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k\") pod \"a2003c14-59bd-4497-9988-f52f04cfff1b\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") "
Jul 07 14:22:24.054810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.054772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location\") pod \"a2003c14-59bd-4497-9988-f52f04cfff1b\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") "
Jul 07 14:22:24.054810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.054792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert\") pod \"a2003c14-59bd-4497-9988-f52f04cfff1b\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") "
Jul 07 14:22:24.055071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.054828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config\") pod \"a2003c14-59bd-4497-9988-f52f04cfff1b\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") "
Jul 07 14:22:24.055071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.054917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls\") pod \"a2003c14-59bd-4497-9988-f52f04cfff1b\" (UID: \"a2003c14-59bd-4497-9988-f52f04cfff1b\") "
Jul 07 14:22:24.055211 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.055162    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2003c14-59bd-4497-9988-f52f04cfff1b" (UID: "a2003c14-59bd-4497-9988-f52f04cfff1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:22:24.055268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.055219    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2003c14-59bd-4497-9988-f52f04cfff1b" (UID: "a2003c14-59bd-4497-9988-f52f04cfff1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:22:24.055268 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.055238    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2003c14-59bd-4497-9988-f52f04cfff1b" (UID: "a2003c14-59bd-4497-9988-f52f04cfff1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:22:24.056928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.056892    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2003c14-59bd-4497-9988-f52f04cfff1b" (UID: "a2003c14-59bd-4497-9988-f52f04cfff1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:22:24.057019 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.056947    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k" (OuterVolumeSpecName: "kube-api-access-k8x4k") pod "a2003c14-59bd-4497-9988-f52f04cfff1b" (UID: "a2003c14-59bd-4497-9988-f52f04cfff1b"). InnerVolumeSpecName "kube-api-access-k8x4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:22:24.088005 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.087981    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" path="/var/lib/kubelet/pods/a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1/volumes"
Jul 07 14:22:24.136678 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.136656    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerID="4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416" exitCode=0
Jul 07 14:22:24.137002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.136730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerDied","Data":"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"}
Jul 07 14:22:24.139510 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.139489    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerID="5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7" exitCode=0
Jul 07 14:22:24.139587 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.139542    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"
Jul 07 14:22:24.139587 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.139569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerDied","Data":"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"}
Jul 07 14:22:24.139704 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.139628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj" event={"ID":"a2003c14-59bd-4497-9988-f52f04cfff1b","Type":"ContainerDied","Data":"5805f11bbce610473cbbf014c789e99e80d1c029f3c1e145e0b2ab6605e69d0c"}
Jul 07 14:22:24.139704 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.139669    2563 scope.go:117] "RemoveContainer" containerID="bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"
Jul 07 14:22:24.141602 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.141576    2563 generic.go:356] "Generic (PLEG): container finished" podID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerID="1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697" exitCode=0
Jul 07 14:22:24.141710 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.141631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerDied","Data":"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"}
Jul 07 14:22:24.150000 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.149986    2563 scope.go:117] "RemoveContainer" containerID="5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"
Jul 07 14:22:24.156124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.156016    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8x4k\" (UniqueName: \"kubernetes.io/projected/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-api-access-k8x4k\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:24.156124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.156043    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2003c14-59bd-4497-9988-f52f04cfff1b-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:24.156124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.156058    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:24.156124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.156071    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2003c14-59bd-4497-9988-f52f04cfff1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:24.156124 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.156086    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2003c14-59bd-4497-9988-f52f04cfff1b-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:22:24.159330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.158786    2563 scope.go:117] "RemoveContainer" containerID="e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"
Jul 07 14:22:24.168433 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.168413    2563 scope.go:117] "RemoveContainer" containerID="bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"
Jul 07 14:22:24.168704 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:24.168685    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de\": container with ID starting with bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de not found: ID does not exist" containerID="bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"
Jul 07 14:22:24.168795 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.168709    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de"} err="failed to get container status \"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de\": rpc error: code = NotFound desc = could not find container \"bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de\": container with ID starting with bc4026501b462fcea15c40ebf20a0dc37fad8b501cf453cec12f889f6437c0de not found: ID does not exist"
Jul 07 14:22:24.168795 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.168724    2563 scope.go:117] "RemoveContainer" containerID="5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"
Jul 07 14:22:24.169019 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:24.168995    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7\": container with ID starting with 5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7 not found: ID does not exist" containerID="5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"
Jul 07 14:22:24.169078 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.169029    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7"} err="failed to get container status \"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7\": rpc error: code = NotFound desc = could not find container \"5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7\": container with ID starting with 5236e87c7b404305e1569c328cb68611fa1592c27fe66fbf39b335d42739d0c7 not found: ID does not exist"
Jul 07 14:22:24.169078 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.169052    2563 scope.go:117] "RemoveContainer" containerID="e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"
Jul 07 14:22:24.169336 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:22:24.169320    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b\": container with ID starting with e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b not found: ID does not exist" containerID="e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"
Jul 07 14:22:24.169391 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.169341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b"} err="failed to get container status \"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b\": rpc error: code = NotFound desc = could not find container \"e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b\": container with ID starting with e2150d9ba7d24918c5f1458973c6376b1676ca88daeb2a25d02f18f30a0b3a0b not found: ID does not exist"
Jul 07 14:22:24.195880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.195860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:22:24.202531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:24.202510    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-884de-predictor-7678456578-zh5nj"]
Jul 07 14:22:25.148135 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.148086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerStarted","Data":"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"}
Jul 07 14:22:25.148617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.148171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerStarted","Data":"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"}
Jul 07 14:22:25.148617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.148366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:25.150892 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.150866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerStarted","Data":"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"}
Jul 07 14:22:25.150999 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.150916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerStarted","Data":"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"}
Jul 07 14:22:25.151123 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.151108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:25.175874 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.175820    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podStartSLOduration=2.477138892 podStartE2EDuration="6.175802076s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698663184s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:22:25.174859305 +0000 UTC m=+739.731295680" watchObservedRunningTime="2026-07-07 14:22:25.175802076 +0000 UTC m=+739.732238440"
Jul 07 14:22:25.201662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:25.201617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podStartSLOduration=2.113846425 podStartE2EDuration="6.201603967s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.087757542s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:22:25.20012182 +0000 UTC m=+739.756558194" watchObservedRunningTime="2026-07-07 14:22:25.201603967 +0000 UTC m=+739.758040331"
Jul 07 14:22:26.089620 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:26.089585    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" path="/var/lib/kubelet/pods/a2003c14-59bd-4497-9988-f52f04cfff1b/volumes"
Jul 07 14:22:26.154405 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:26.154375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:26.154813 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:26.154437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:26.155395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:26.155359    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:22:26.155525 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:26.155490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:22:27.160228 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:27.160187    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:22:27.160641 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:27.160284    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:22:32.164858 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:32.164814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:22:32.165752 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:32.165730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:22:32.165888 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:32.165801    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:22:32.166237 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:32.166210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:22:42.166061 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:42.166017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:22:42.166539 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:42.166168    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:22:52.166206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:52.166170    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:22:52.166206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:22:52.166191    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:23:02.166121 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:02.166053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:23:02.166551 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:02.166224    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:23:12.166308 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:12.166269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:23:12.166690 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:12.166333    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:23:22.165908 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:22.165870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:23:22.166506 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:22.166080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:23:32.166313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:32.166229    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:23:32.166876 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:32.166857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:23:42.166991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:42.166961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:23:59.229927 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:59.229892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:23:59.367457 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:23:59.367415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:09.290788 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.290759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:09.428452 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.428417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:09.487549 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.487512    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:24:09.487903 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.487869    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" containerID="cri-o://afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0" gracePeriod=30
Jul 07 14:24:09.488047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.487968    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kube-rbac-proxy" containerID="cri-o://28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77" gracePeriod=30
Jul 07 14:24:09.533928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.533891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:09.534841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.534822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:24:09.535071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.535050    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" containerID="cri-o://e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff" gracePeriod=30
Jul 07 14:24:09.535149 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.535090    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kube-rbac-proxy" containerID="cri-o://12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832" gracePeriod=30
Jul 07 14:24:09.632451 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:09.632412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:10.571886 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:10.571850    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerID="28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77" exitCode=2
Jul 07 14:24:10.572376 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:10.571918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerDied","Data":"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"}
Jul 07 14:24:10.574149 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:10.574123    2563 generic.go:356] "Generic (PLEG): container finished" podID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerID="12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832" exitCode=2
Jul 07 14:24:10.574303 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:10.574157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerDied","Data":"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"}
Jul 07 14:24:12.160788 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:12.160746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 14:24:12.160788 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:12.160761    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 07 14:24:12.166577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:12.166538    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:24:12.166577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:12.166538    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 14:24:13.181415 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.181390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:24:13.242768 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.242695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config\") pod \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") "
Jul 07 14:24:13.242768 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.242728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert\") pod \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") "
Jul 07 14:24:13.242768 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.242752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq6pt\" (UniqueName: \"kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt\") pod \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") "
Jul 07 14:24:13.243028 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.242791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls\") pod \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") "
Jul 07 14:24:13.243028 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.242808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location\") pod \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\" (UID: \"e95a58c6-8d8c-4249-80db-35ef7ca89c2d\") "
Jul 07 14:24:13.243216 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.243186    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e95a58c6-8d8c-4249-80db-35ef7ca89c2d" (UID: "e95a58c6-8d8c-4249-80db-35ef7ca89c2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:13.243327 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.243190    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e95a58c6-8d8c-4249-80db-35ef7ca89c2d" (UID: "e95a58c6-8d8c-4249-80db-35ef7ca89c2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:13.243327 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.243220    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e95a58c6-8d8c-4249-80db-35ef7ca89c2d" (UID: "e95a58c6-8d8c-4249-80db-35ef7ca89c2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:24:13.244880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.244860    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e95a58c6-8d8c-4249-80db-35ef7ca89c2d" (UID: "e95a58c6-8d8c-4249-80db-35ef7ca89c2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:24:13.244974 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.244944    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt" (OuterVolumeSpecName: "kube-api-access-hq6pt") pod "e95a58c6-8d8c-4249-80db-35ef7ca89c2d" (UID: "e95a58c6-8d8c-4249-80db-35ef7ca89c2d"). InnerVolumeSpecName "kube-api-access-hq6pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:24:13.343778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.343736    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:13.343778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.343774    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:13.343778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.343784    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq6pt\" (UniqueName: \"kubernetes.io/projected/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kube-api-access-hq6pt\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:13.344003 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.343793    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:13.344003 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.343802    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e95a58c6-8d8c-4249-80db-35ef7ca89c2d-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:13.587344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.587244    2563 generic.go:356] "Generic (PLEG): container finished" podID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerID="e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff" exitCode=0
Jul 07 14:24:13.587344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.587314    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"
Jul 07 14:24:13.587344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.587333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerDied","Data":"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"}
Jul 07 14:24:13.587609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.587394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz" event={"ID":"e95a58c6-8d8c-4249-80db-35ef7ca89c2d","Type":"ContainerDied","Data":"42bd767f382ba6894bc118ac1d8a3faf6c01bafb201ab0e359c635206dcf83c8"}
Jul 07 14:24:13.587609 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.587421    2563 scope.go:117] "RemoveContainer" containerID="12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"
Jul 07 14:24:13.595881 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.595868    2563 scope.go:117] "RemoveContainer" containerID="e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"
Jul 07 14:24:13.602956 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.602940    2563 scope.go:117] "RemoveContainer" containerID="1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"
Jul 07 14:24:13.612796 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.612779    2563 scope.go:117] "RemoveContainer" containerID="12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"
Jul 07 14:24:13.613165 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:13.613144    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832\": container with ID starting with 12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832 not found: ID does not exist" containerID="12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"
Jul 07 14:24:13.613240 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613172    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832"} err="failed to get container status \"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832\": rpc error: code = NotFound desc = could not find container \"12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832\": container with ID starting with 12736c54453f557e617a1f4abdc2fb68ce3e6628d0096029c750b7436482a832 not found: ID does not exist"
Jul 07 14:24:13.613240 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613189    2563 scope.go:117] "RemoveContainer" containerID="e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"
Jul 07 14:24:13.613459 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:13.613439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff\": container with ID starting with e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff not found: ID does not exist" containerID="e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"
Jul 07 14:24:13.613506 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff"} err="failed to get container status \"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff\": rpc error: code = NotFound desc = could not find container \"e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff\": container with ID starting with e5c363f2069a2adff271638a86857f1015bf66cbc98bcb9f942253af890899ff not found: ID does not exist"
Jul 07 14:24:13.613506 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613487    2563 scope.go:117] "RemoveContainer" containerID="1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"
Jul 07 14:24:13.613753 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:13.613736    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697\": container with ID starting with 1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697 not found: ID does not exist" containerID="1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"
Jul 07 14:24:13.613810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613761    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697"} err="failed to get container status \"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697\": rpc error: code = NotFound desc = could not find container \"1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697\": container with ID starting with 1223d2c9b7cc179d5bbfcee6b36c7067c63f13900e0a397aadf304ae0c81e697 not found: ID does not exist"
Jul 07 14:24:13.613810 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.613791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:24:13.615119 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:13.615085    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd33a-predictor-6d98dbdfd9-slvbz"]
Jul 07 14:24:14.089962 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.089936    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" path="/var/lib/kubelet/pods/e95a58c6-8d8c-4249-80db-35ef7ca89c2d/volumes"
Jul 07 14:24:14.133380 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.133360    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:24:14.249752 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.249670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kg8v\" (UniqueName: \"kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v\") pod \"7fe340e8-5229-4970-b432-41feb1f12ed2\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") "
Jul 07 14:24:14.249752 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.249725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config\") pod \"7fe340e8-5229-4970-b432-41feb1f12ed2\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") "
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.249757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location\") pod \"7fe340e8-5229-4970-b432-41feb1f12ed2\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") "
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.249809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert\") pod \"7fe340e8-5229-4970-b432-41feb1f12ed2\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") "
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.249868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls\") pod \"7fe340e8-5229-4970-b432-41feb1f12ed2\" (UID: \"7fe340e8-5229-4970-b432-41feb1f12ed2\") "
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.250143    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fe340e8-5229-4970-b432-41feb1f12ed2" (UID: "7fe340e8-5229-4970-b432-41feb1f12ed2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.250161    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fe340e8-5229-4970-b432-41feb1f12ed2" (UID: "7fe340e8-5229-4970-b432-41feb1f12ed2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:14.250289 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.250249    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fe340e8-5229-4970-b432-41feb1f12ed2" (UID: "7fe340e8-5229-4970-b432-41feb1f12ed2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:14.251970 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.251946    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fe340e8-5229-4970-b432-41feb1f12ed2" (UID: "7fe340e8-5229-4970-b432-41feb1f12ed2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:24:14.252232 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.252210    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v" (OuterVolumeSpecName: "kube-api-access-4kg8v") pod "7fe340e8-5229-4970-b432-41feb1f12ed2" (UID: "7fe340e8-5229-4970-b432-41feb1f12ed2"). InnerVolumeSpecName "kube-api-access-4kg8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:24:14.351047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.351014    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:14.351047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.351041    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fe340e8-5229-4970-b432-41feb1f12ed2-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:14.351047 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.351051    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg8v\" (UniqueName: \"kubernetes.io/projected/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-api-access-4kg8v\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:14.351297 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.351062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fe340e8-5229-4970-b432-41feb1f12ed2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:14.351297 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.351071    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe340e8-5229-4970-b432-41feb1f12ed2-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:24:14.594429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.593133    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerID="afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0" exitCode=0
Jul 07 14:24:14.594429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.593317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerDied","Data":"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"}
Jul 07 14:24:14.594429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.593427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9" event={"ID":"7fe340e8-5229-4970-b432-41feb1f12ed2","Type":"ContainerDied","Data":"46fccb341f8df27158d3b73a9f205bce1ea40065cc81cd12c59c4a3f518d2d0e"}
Jul 07 14:24:14.594429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.593480    2563 scope.go:117] "RemoveContainer" containerID="28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"
Jul 07 14:24:14.594429 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.593726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"
Jul 07 14:24:14.606160 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.606143    2563 scope.go:117] "RemoveContainer" containerID="afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"
Jul 07 14:24:14.613572 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.613557    2563 scope.go:117] "RemoveContainer" containerID="4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"
Jul 07 14:24:14.618945 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.618924    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:24:14.620972 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.620891    2563 scope.go:117] "RemoveContainer" containerID="28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"
Jul 07 14:24:14.621410 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:14.621386    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77\": container with ID starting with 28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77 not found: ID does not exist" containerID="28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"
Jul 07 14:24:14.621545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.621421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77"} err="failed to get container status \"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77\": rpc error: code = NotFound desc = could not find container \"28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77\": container with ID starting with 28da20e5358c8fc370f99d6f2b3d4c1d20eff1dce4d02afd16da1b4e5bea8f77 not found: ID does not exist"
Jul 07 14:24:14.621545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.621446    2563 scope.go:117] "RemoveContainer" containerID="afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"
Jul 07 14:24:14.621735 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:14.621718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0\": container with ID starting with afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0 not found: ID does not exist" containerID="afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"
Jul 07 14:24:14.621778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.621743    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0"} err="failed to get container status \"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0\": rpc error: code = NotFound desc = could not find container \"afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0\": container with ID starting with afff91c24cdaff33fe0887921ebd812c472af5a12ad0c4cc09154dee4f4c16b0 not found: ID does not exist"
Jul 07 14:24:14.621778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.621758    2563 scope.go:117] "RemoveContainer" containerID="4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"
Jul 07 14:24:14.622028 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:24:14.622005    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416\": container with ID starting with 4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416 not found: ID does not exist" containerID="4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"
Jul 07 14:24:14.622081 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.622036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416"} err="failed to get container status \"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416\": rpc error: code = NotFound desc = could not find container \"4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416\": container with ID starting with 4331a6deb8f80e35dfa2df722ce66ee2580fbe78724cdbaeb5c4b94992ab4416 not found: ID does not exist"
Jul 07 14:24:14.622720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:14.622705    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd33a-predictor-5f66864c98-c6lp9"]
Jul 07 14:24:16.088745 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:16.088711    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" path="/var/lib/kubelet/pods/7fe340e8-5229-4970-b432-41feb1f12ed2/volumes"
Jul 07 14:24:29.505745 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.505712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:29.515089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:24:29.515403 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container"
Jul 07 14:24:29.515448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515406    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container"
Jul 07 14:24:29.515448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515416    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515425    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="storage-initializer"
Jul 07 14:24:29.515448 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515450    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515456    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515468    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515496    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515519    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515524    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="storage-initializer"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515546    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kube-rbac-proxy"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container"
Jul 07 14:24:29.515595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515560    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kube-rbac-proxy"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515618    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kserve-container"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515626    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03cc6cd-e0fc-4959-866a-b7d7d3d13cd1" containerName="kserve-container"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kube-rbac-proxy"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kube-rbac-proxy"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515644    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2003c14-59bd-4497-9988-f52f04cfff1b" containerName="kserve-container"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe340e8-5229-4970-b432-41feb1f12ed2" containerName="kserve-container"
Jul 07 14:24:29.516120 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.515660    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95a58c6-8d8c-4249-80db-35ef7ca89c2d" containerName="kube-rbac-proxy"
Jul 07 14:24:29.520423 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.520406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.523233 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:24:29.523481 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:24:29.523577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523505    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:24:29.523577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-7a71c-predictor-serving-cert\""
Jul 07 14:24:29.523577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523544    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:24:29.523577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-7a71c-kube-rbac-proxy-sar-config\""
Jul 07 14:24:29.523964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.523951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8rhzg\""
Jul 07 14:24:29.535878 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.535860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:24:29.673139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.673092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.673293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.673170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhzm\" (UniqueName: \"kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.673293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.673224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.673293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.673277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.673422 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.673334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.683138 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.683115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:24:29.774002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.773935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.773974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774214 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhzm\" (UniqueName: \"kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774214 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774311 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774533 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774513    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774774 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.774815 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.774779    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.776302 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.776284    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.783053 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.783034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhzm\" (UniqueName: \"kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm\") pod \"isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.831108 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.831086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:29.957474 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:29.957453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:24:29.959315 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:24:29.959289    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fc836e_1216_4e91_9ef9_19c2043f8178.slice/crio-67788ce690b70e2b8d3372b00a9d96497b290ac71c2f1650236c0a886f0189e3 WatchSource:0}: Error finding container 67788ce690b70e2b8d3372b00a9d96497b290ac71c2f1650236c0a886f0189e3: Status 404 returned error can't find the container with id 67788ce690b70e2b8d3372b00a9d96497b290ac71c2f1650236c0a886f0189e3
Jul 07 14:24:30.659632 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:30.659602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerStarted","Data":"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"}
Jul 07 14:24:30.660080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:30.659665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerStarted","Data":"67788ce690b70e2b8d3372b00a9d96497b290ac71c2f1650236c0a886f0189e3"}
Jul 07 14:24:34.679841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:34.679806    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerID="e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28" exitCode=0
Jul 07 14:24:34.680348 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:34.679887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerDied","Data":"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"}
Jul 07 14:24:35.686041 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.686009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerStarted","Data":"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"}
Jul 07 14:24:35.686430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.686079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerStarted","Data":"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"}
Jul 07 14:24:35.686430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.686131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerStarted","Data":"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"}
Jul 07 14:24:35.686430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.686423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:35.686531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.686470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:35.687880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.687854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:24:35.710807 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:35.710765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podStartSLOduration=2.981198515 podStartE2EDuration="6.71075196s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729553445s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:35.709331645 +0000 UTC m=+870.265768053" watchObservedRunningTime="2026-07-07 14:24:35.71075196 +0000 UTC m=+870.267188324"
Jul 07 14:24:36.690994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:36.690957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:36.691476 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:36.691072    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:24:36.692397 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:36.692373    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:37.695096 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:37.695046    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:24:37.695531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:37.695511    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:42.700048 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:42.700015    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:24:42.700671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:42.700642    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:24:42.701153 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:42.701125    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:52.700695 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:52.700647    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:24:52.701164 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:24:52.701069    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:02.700743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:02.700644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:25:02.701158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:02.701057    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:12.700955 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:12.700910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:25:12.701409 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:12.701389    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:22.700708 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:22.700659    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:25:22.701200 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:22.701091    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:32.701407 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:32.701367    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:25:32.701916 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:32.701891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:42.700909 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:42.700866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:25:42.703408 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:42.701379    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:52.701607 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:52.701574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:25:52.702070 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:52.701684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:25:56.735870 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:56.735827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:25:56.821549 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:25:56.821518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:26:04.811349 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.811311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:26:04.958048 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.957975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:26:04.958279 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.958254    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" containerID="cri-o://c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d" gracePeriod=30
Jul 07 14:26:04.958361 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.958274    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" containerID="cri-o://08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9" gracePeriod=30
Jul 07 14:26:04.958430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.958317    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" containerID="cri-o://be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b" gracePeriod=30
Jul 07 14:26:04.987574 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.987550    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:26:04.991112 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.991082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:04.996491 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.996470    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-9938b-kube-rbac-proxy-sar-config\""
Jul 07 14:26:04.996615 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:04.996470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-9938b-predictor-serving-cert\""
Jul 07 14:26:05.020319 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.020290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:26:05.047680 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.047652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.047924 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.047730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.047924 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.047750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.047924 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.047873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.047924 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.047914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz5j\" (UniqueName: \"kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.060459 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.060431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:26:05.148446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.148446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.148686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.148686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz5j\" (UniqueName: \"kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.148686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.148974 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.148952    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.149192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.149175    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.149248 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.149229    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.151091 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.151071    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.156845 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.156808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:26:05.160574 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.160556    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz5j\" (UniqueName: \"kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j\") pod \"isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.301962 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.301930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:05.438458 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:05.438280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:26:05.441013 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:26:05.440986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d182cc8_09e3_46d9_9445_1b2f2eb9fd3e.slice/crio-cd5fb43fae6aadb5c7a7535ebce1fd4cf72a8f56f60e07c2702818c9fb0a43bc WatchSource:0}: Error finding container cd5fb43fae6aadb5c7a7535ebce1fd4cf72a8f56f60e07c2702818c9fb0a43bc: Status 404 returned error can't find the container with id cd5fb43fae6aadb5c7a7535ebce1fd4cf72a8f56f60e07c2702818c9fb0a43bc
Jul 07 14:26:06.045634 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:06.045599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerStarted","Data":"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"}
Jul 07 14:26:06.046049 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:06.045657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerStarted","Data":"cd5fb43fae6aadb5c7a7535ebce1fd4cf72a8f56f60e07c2702818c9fb0a43bc"}
Jul 07 14:26:06.047865 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:06.047844    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerID="be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b" exitCode=2
Jul 07 14:26:06.047971 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:06.047916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerDied","Data":"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"}
Jul 07 14:26:07.695669 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:07.695627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:10.065951 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:10.065919    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerID="bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34" exitCode=0
Jul 07 14:26:10.066395 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:10.065990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerDied","Data":"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"}
Jul 07 14:26:10.068469 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:10.068449    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerID="c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d" exitCode=0
Jul 07 14:26:10.068574 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:10.068474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerDied","Data":"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"}
Jul 07 14:26:11.074142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:11.074090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerStarted","Data":"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"}
Jul 07 14:26:11.074541 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:11.074174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerStarted","Data":"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"}
Jul 07 14:26:11.074541 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:11.074410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:11.098778 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:11.098733    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podStartSLOduration=3.218709618 podStartE2EDuration="7.098719076s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880009458s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:26:11.096145585 +0000 UTC m=+965.652581950" watchObservedRunningTime="2026-07-07 14:26:11.098719076 +0000 UTC m=+965.655155499"
Jul 07 14:26:12.077857 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:12.077822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:12.079074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:12.079045    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:12.695340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:12.695299    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:12.700702 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:12.700662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:26:12.700965 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:12.700945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:13.082340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:13.082246    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:17.695878 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:17.695837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:17.696273 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:17.695968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:26:18.088756 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:18.088681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:26:18.089151 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:18.089116    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:22.696084 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:22.696043    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:22.700769 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:22.700743    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:26:22.701168 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:22.701146    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:27.695742 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:27.695644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:28.089426 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:28.089343    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:32.695741 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:32.695699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 14:26:32.701089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:32.701063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 14:26:32.701217 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:32.701203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:26:32.701442 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:32.701412    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:32.701524 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:32.701513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:26:35.117042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.117017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:26:35.167692 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.167661    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerID="08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9" exitCode=0
Jul 07 14:26:35.167840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.167706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerDied","Data":"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"}
Jul 07 14:26:35.167840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.167725    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"
Jul 07 14:26:35.167840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.167766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5" event={"ID":"c8fc836e-1216-4e91-9ef9-19c2043f8178","Type":"ContainerDied","Data":"67788ce690b70e2b8d3372b00a9d96497b290ac71c2f1650236c0a886f0189e3"}
Jul 07 14:26:35.167840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.167798    2563 scope.go:117] "RemoveContainer" containerID="08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"
Jul 07 14:26:35.175735 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.175713    2563 scope.go:117] "RemoveContainer" containerID="be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"
Jul 07 14:26:35.182820 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.182806    2563 scope.go:117] "RemoveContainer" containerID="c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"
Jul 07 14:26:35.189695 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.189667    2563 scope.go:117] "RemoveContainer" containerID="e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"
Jul 07 14:26:35.196375 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.196360    2563 scope.go:117] "RemoveContainer" containerID="08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"
Jul 07 14:26:35.196623 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:26:35.196605    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9\": container with ID starting with 08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9 not found: ID does not exist" containerID="08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"
Jul 07 14:26:35.196675 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.196640    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9"} err="failed to get container status \"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9\": rpc error: code = NotFound desc = could not find container \"08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9\": container with ID starting with 08d6fda260bae475eb17d700c744ccf1802137f74a239f30d93c8c9d74b817f9 not found: ID does not exist"
Jul 07 14:26:35.196675 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.196657    2563 scope.go:117] "RemoveContainer" containerID="be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"
Jul 07 14:26:35.196908 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:26:35.196890    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b\": container with ID starting with be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b not found: ID does not exist" containerID="be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"
Jul 07 14:26:35.196948 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.196915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b"} err="failed to get container status \"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b\": rpc error: code = NotFound desc = could not find container \"be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b\": container with ID starting with be6e358bc4ba5cb9911cbc6a52050c2a6d14cea385c5a6646419cdf9ff74b97b not found: ID does not exist"
Jul 07 14:26:35.196948 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.196931    2563 scope.go:117] "RemoveContainer" containerID="c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"
Jul 07 14:26:35.197166 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:26:35.197148    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d\": container with ID starting with c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d not found: ID does not exist" containerID="c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"
Jul 07 14:26:35.197226 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.197171    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d"} err="failed to get container status \"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d\": rpc error: code = NotFound desc = could not find container \"c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d\": container with ID starting with c3721b93cffccbd4a85ba35bdbb26a9feb9b41d9b0a3b2036d8b7866effa845d not found: ID does not exist"
Jul 07 14:26:35.197226 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.197185    2563 scope.go:117] "RemoveContainer" containerID="e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"
Jul 07 14:26:35.197425 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:26:35.197409    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28\": container with ID starting with e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28 not found: ID does not exist" containerID="e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"
Jul 07 14:26:35.197473 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.197428    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28"} err="failed to get container status \"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28\": rpc error: code = NotFound desc = could not find container \"e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28\": container with ID starting with e017e157c12a483c69e1344a0c5a33a5928402357b07232d03ed634737fc5c28 not found: ID does not exist"
Jul 07 14:26:35.307915 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.307839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqhzm\" (UniqueName: \"kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm\") pod \"c8fc836e-1216-4e91-9ef9-19c2043f8178\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") "
Jul 07 14:26:35.307915 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.307899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location\") pod \"c8fc836e-1216-4e91-9ef9-19c2043f8178\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") "
Jul 07 14:26:35.308122 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.307918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert\") pod \"c8fc836e-1216-4e91-9ef9-19c2043f8178\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") "
Jul 07 14:26:35.308122 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.307956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config\") pod \"c8fc836e-1216-4e91-9ef9-19c2043f8178\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") "
Jul 07 14:26:35.308122 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.307978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls\") pod \"c8fc836e-1216-4e91-9ef9-19c2043f8178\" (UID: \"c8fc836e-1216-4e91-9ef9-19c2043f8178\") "
Jul 07 14:26:35.308311 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.308288    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8fc836e-1216-4e91-9ef9-19c2043f8178" (UID: "c8fc836e-1216-4e91-9ef9-19c2043f8178"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:26:35.308311 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.308300    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8fc836e-1216-4e91-9ef9-19c2043f8178" (UID: "c8fc836e-1216-4e91-9ef9-19c2043f8178"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:35.308419 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.308347    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8fc836e-1216-4e91-9ef9-19c2043f8178" (UID: "c8fc836e-1216-4e91-9ef9-19c2043f8178"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:35.309848 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.309816    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm" (OuterVolumeSpecName: "kube-api-access-kqhzm") pod "c8fc836e-1216-4e91-9ef9-19c2043f8178" (UID: "c8fc836e-1216-4e91-9ef9-19c2043f8178"). InnerVolumeSpecName "kube-api-access-kqhzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:26:35.309956 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.309863    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8fc836e-1216-4e91-9ef9-19c2043f8178" (UID: "c8fc836e-1216-4e91-9ef9-19c2043f8178"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:26:35.408922 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.408894    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:26:35.408922 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.408920    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fc836e-1216-4e91-9ef9-19c2043f8178-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:26:35.409135 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.408934    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqhzm\" (UniqueName: \"kubernetes.io/projected/c8fc836e-1216-4e91-9ef9-19c2043f8178-kube-api-access-kqhzm\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:26:35.409135 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.408942    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fc836e-1216-4e91-9ef9-19c2043f8178-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:26:35.409135 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.408951    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fc836e-1216-4e91-9ef9-19c2043f8178-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:26:35.492568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.492537    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:26:35.496258 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:35.496231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7a71c-predictor-9798d7c8b-df5l5"]
Jul 07 14:26:36.087804 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:36.087777    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" path="/var/lib/kubelet/pods/c8fc836e-1216-4e91-9ef9-19c2043f8178/volumes"
Jul 07 14:26:38.089355 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:38.089324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:48.090038 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:48.089987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:26:58.089321 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:26:58.089281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:08.093015 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:08.092977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:08.754980 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:08.754943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:27:18.089519 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:18.089476    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:28.090134 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:28.090080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:31.083667 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:31.083627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:41.084023 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:41.083974    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:27:51.084301 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:27:51.084255    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:01.083886 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:01.083789    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:11.084305 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:11.084260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:21.084582 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:21.084542    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:31.084271 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:31.084227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:41.084305 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:41.084270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:28:44.991928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:44.991897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:28:44.992413 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:44.992217    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" containerID="cri-o://c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a" gracePeriod=30
Jul 07 14:28:44.992413 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:44.992216    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy" containerID="cri-o://e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099" gracePeriod=30
Jul 07 14:28:45.063964 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:45.063923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:28:45.679080 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:45.679041    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerID="e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099" exitCode=2
Jul 07 14:28:45.679269 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:45.679116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerDied","Data":"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"}
Jul 07 14:28:48.083154 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:48.083079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 14:28:50.044502 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.044462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:28:50.098399 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098364    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:28:50.098795 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="storage-initializer"
Jul 07 14:28:50.098795 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098795    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="storage-initializer"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098820    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098828    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098851    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098859    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098875    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098883    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098965    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="agent"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098978    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kube-rbac-proxy"
Jul 07 14:28:50.099008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.098988    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8fc836e-1216-4e91-9ef9-19c2043f8178" containerName="kserve-container"
Jul 07 14:28:50.102330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.102309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.104955 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.104936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-07975f-kube-rbac-proxy-sar-config\""
Jul 07 14:28:50.105262 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.105249    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-07975f-predictor-serving-cert\""
Jul 07 14:28:50.114901 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.114873    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:28:50.139747 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.139722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:28:50.192029 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.192002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.192165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.192045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.192165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.192088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.192165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.192157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dp4\" (UniqueName: \"kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.192330 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.192199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.292846 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.292823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dp4\" (UniqueName: \"kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.292959 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.292858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293027 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.292984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293084 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.293028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293084 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.293054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293257 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.293237    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293763 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.293742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.293806 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.293742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.295341 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.295300    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.302517 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.302491    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dp4\" (UniqueName: \"kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4\") pod \"isvc-primary-07975f-predictor-8574dbdbd7-fvkgg\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") " pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.412520 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.412495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:50.540867 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.540845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:28:50.542752 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:28:50.542727    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37620da4_15da_4d0d_ad8f_e404d3b8e339.slice/crio-e86fd084a1cf7d4aa6d42388b7080c922142e6aa2aa2cca01a62f1163b062e9c WatchSource:0}: Error finding container e86fd084a1cf7d4aa6d42388b7080c922142e6aa2aa2cca01a62f1163b062e9c: Status 404 returned error can't find the container with id e86fd084a1cf7d4aa6d42388b7080c922142e6aa2aa2cca01a62f1163b062e9c
Jul 07 14:28:50.700196 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.700157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerStarted","Data":"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"}
Jul 07 14:28:50.700351 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:50.700218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerStarted","Data":"e86fd084a1cf7d4aa6d42388b7080c922142e6aa2aa2cca01a62f1163b062e9c"}
Jul 07 14:28:51.084018 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:51.083975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:28:53.083313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:53.083270    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 14:28:54.536639 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.536619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:28:54.630776 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.630748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config\") pod \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") "
Jul 07 14:28:54.630946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.630787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert\") pod \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") "
Jul 07 14:28:54.630946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.630806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location\") pod \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") "
Jul 07 14:28:54.630946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.630844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltz5j\" (UniqueName: \"kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j\") pod \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") "
Jul 07 14:28:54.630946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.630870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls\") pod \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\" (UID: \"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e\") "
Jul 07 14:28:54.631206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.631158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" (UID: "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:28:54.631206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.631171    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" (UID: "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:28:54.631206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.631183    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" (UID: "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:28:54.632840 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.632812    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j" (OuterVolumeSpecName: "kube-api-access-ltz5j") pod "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" (UID: "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e"). InnerVolumeSpecName "kube-api-access-ltz5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:28:54.632932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.632911    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" (UID: "9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:28:54.716786 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.716762    2563 generic.go:356] "Generic (PLEG): container finished" podID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerID="e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3" exitCode=0
Jul 07 14:28:54.716912 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.716836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerDied","Data":"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"}
Jul 07 14:28:54.718716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.718695    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerID="c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a" exitCode=0
Jul 07 14:28:54.718805 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.718742    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"
Jul 07 14:28:54.718805 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.718762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerDied","Data":"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"}
Jul 07 14:28:54.718884 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.718805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8" event={"ID":"9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e","Type":"ContainerDied","Data":"cd5fb43fae6aadb5c7a7535ebce1fd4cf72a8f56f60e07c2702818c9fb0a43bc"}
Jul 07 14:28:54.718884 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.718834    2563 scope.go:117] "RemoveContainer" containerID="e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"
Jul 07 14:28:54.730167 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.730148    2563 scope.go:117] "RemoveContainer" containerID="c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"
Jul 07 14:28:54.732206 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.732187    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltz5j\" (UniqueName: \"kubernetes.io/projected/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-api-access-ltz5j\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:28:54.732292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.732211    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:28:54.732292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.732226    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:28:54.732292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.732241    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:28:54.732292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.732253    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:28:54.737216 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.737200    2563 scope.go:117] "RemoveContainer" containerID="bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"
Jul 07 14:28:54.750559 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.750539    2563 scope.go:117] "RemoveContainer" containerID="e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"
Jul 07 14:28:54.750808 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:28:54.750789    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099\": container with ID starting with e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099 not found: ID does not exist" containerID="e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"
Jul 07 14:28:54.750886 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.750814    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099"} err="failed to get container status \"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099\": rpc error: code = NotFound desc = could not find container \"e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099\": container with ID starting with e62008312a67c88ed249f14d2562c09e6e487a0f1f27072b09c7b4719714c099 not found: ID does not exist"
Jul 07 14:28:54.750886 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.750830    2563 scope.go:117] "RemoveContainer" containerID="c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"
Jul 07 14:28:54.751080 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:28:54.751064    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a\": container with ID starting with c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a not found: ID does not exist" containerID="c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"
Jul 07 14:28:54.751165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.751084    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a"} err="failed to get container status \"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a\": rpc error: code = NotFound desc = could not find container \"c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a\": container with ID starting with c99cf24f21b8d48165a14c35b470aa71b9bbbe7eaceefdecf2c598f370e2b34a not found: ID does not exist"
Jul 07 14:28:54.751165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.751111    2563 scope.go:117] "RemoveContainer" containerID="bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"
Jul 07 14:28:54.751355 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:28:54.751336    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34\": container with ID starting with bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34 not found: ID does not exist" containerID="bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"
Jul 07 14:28:54.751422 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.751357    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34"} err="failed to get container status \"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34\": rpc error: code = NotFound desc = could not find container \"bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34\": container with ID starting with bc4e108f4949043645e8ece6ba6ccb8856c01a35d4f0f1c62318f27dc66a3a34 not found: ID does not exist"
Jul 07 14:28:54.755347 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.755328    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:28:54.761628 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:54.761606    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9938b-predictor-7c66754bb4-ms4z8"]
Jul 07 14:28:55.725279 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:55.725245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerStarted","Data":"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"}
Jul 07 14:28:55.725646 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:55.725326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerStarted","Data":"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"}
Jul 07 14:28:55.725646 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:55.725505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:56.088154 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:56.088066    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" path="/var/lib/kubelet/pods/9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e/volumes"
Jul 07 14:28:56.729650 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:56.729626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:28:56.730989 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:56.730964    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:28:57.733817 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:28:57.733777    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:02.738743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:02.738712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:29:02.739253 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:02.739227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:02.762167 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:02.762095    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podStartSLOduration=9.128986672 podStartE2EDuration="12.762077403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633090731s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:28:55.750596058 +0000 UTC m=+1130.307032423" watchObservedRunningTime="2026-07-07 14:29:02.762077403 +0000 UTC m=+1137.318513766"
Jul 07 14:29:05.309893 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:05.309852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:29:05.414666 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:05.414633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:29:12.739498 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:12.739453    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:15.712342 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:15.712288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:29:22.739490 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:22.739451    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:32.739866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:32.739784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:42.743002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:42.742888    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:29:52.739757 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:29:52.739717    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:30:02.739556 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:02.739509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:30:10.087583 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:10.087556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:30:20.304668 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.304615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:20.330248 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330217    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:20.330610 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="storage-initializer"
Jul 07 14:30:20.330610 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="storage-initializer"
Jul 07 14:30:20.330730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330626    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container"
Jul 07 14:30:20.330730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330632    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container"
Jul 07 14:30:20.330730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy"
Jul 07 14:30:20.330730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330655    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy"
Jul 07 14:30:20.330730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330725    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kserve-container"
Jul 07 14:30:20.330885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.330740    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d182cc8-09e3-46d9-9445-1b2f2eb9fd3e" containerName="kube-rbac-proxy"
Jul 07 14:30:20.333853 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.333839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.336606 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.336582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-07975f\""
Jul 07 14:30:20.336734 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.336667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-07975f-dockercfg-mw7fz\""
Jul 07 14:30:20.337032 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.337013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-07975f-kube-rbac-proxy-sar-config\""
Jul 07 14:30:20.337142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.337124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-07975f-predictor-serving-cert\""
Jul 07 14:30:20.346907 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.346885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:20.434509 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.434480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.434671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.434527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.434671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.434564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcbf\" (UniqueName: \"kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.434671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.434607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.434671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.434638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535642 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcbf\" (UniqueName: \"kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535642 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535642 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535642 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.535884 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:20.535686    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-07975f-predictor-serving-cert: secret "isvc-secondary-07975f-predictor-serving-cert" not found
Jul 07 14:30:20.535884 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:20.535757    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls podName:8b6caba9-a942-443b-8dbe-0b462a60d80b nodeName:}" failed. No retries permitted until 2026-07-07 14:30:21.035737556 +0000 UTC m=+1215.592173901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls") pod "isvc-secondary-07975f-predictor-7b8f47b784-gz77k" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b") : secret "isvc-secondary-07975f-predictor-serving-cert" not found
Jul 07 14:30:20.535997 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.535978    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.536313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.536289    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.536406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.536296    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.546696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.546675    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcbf\" (UniqueName: \"kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:20.676042 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:20.676012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:21.040020 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:21.039922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:21.042418 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:21.042390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") pod \"isvc-secondary-07975f-predictor-7b8f47b784-gz77k\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") " pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:21.243879 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:21.243850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:21.368574 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:21.368550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:21.370011 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:30:21.369989    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6caba9_a942_443b_8dbe_0b462a60d80b.slice/crio-480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017 WatchSource:0}: Error finding container 480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017: Status 404 returned error can't find the container with id 480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017
Jul 07 14:30:22.072037 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:22.072003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerStarted","Data":"8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"}
Jul 07 14:30:22.072224 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:22.072058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerStarted","Data":"480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017"}
Jul 07 14:30:25.086314 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:25.086286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/0.log"
Jul 07 14:30:25.086672 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:25.086321    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerID="8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f" exitCode=1
Jul 07 14:30:25.086672 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:25.086396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerDied","Data":"8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"}
Jul 07 14:30:26.090850 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:26.090826    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/0.log"
Jul 07 14:30:26.091248 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:26.090931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerStarted","Data":"9bda96941c328bef8548bd1083887ca414710f2bb47a55eb138c8d7e4e57080f"}
Jul 07 14:30:31.110752 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.110725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/1.log"
Jul 07 14:30:31.111202 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.111134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/0.log"
Jul 07 14:30:31.111202 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.111171    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerID="9bda96941c328bef8548bd1083887ca414710f2bb47a55eb138c8d7e4e57080f" exitCode=1
Jul 07 14:30:31.111313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.111245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerDied","Data":"9bda96941c328bef8548bd1083887ca414710f2bb47a55eb138c8d7e4e57080f"}
Jul 07 14:30:31.111313 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.111304    2563 scope.go:117] "RemoveContainer" containerID="8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"
Jul 07 14:30:31.111613 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.111594    2563 scope.go:117] "RemoveContainer" containerID="8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"
Jul 07 14:30:31.122009 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:31.121988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-07975f-predictor-7b8f47b784-gz77k_kserve-ci-e2e-test(8b6caba9-a942-443b-8dbe-0b462a60d80b)\"" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b"
Jul 07 14:30:31.122193 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:31.122168    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_kserve-ci-e2e-test_8b6caba9-a942-443b-8dbe-0b462a60d80b_0 in pod sandbox 480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017 from index: no such id: '8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f'" containerID="8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"
Jul 07 14:30:31.122273 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:31.122202    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_kserve-ci-e2e-test_8b6caba9-a942-443b-8dbe-0b462a60d80b_0 in pod sandbox 480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017 from index: no such id: '8b92b8ae4944979acacfca7308deff15062cea25696c5d107f3812e465f98c7f'"
Jul 07 14:30:32.115895 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:32.115865    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/1.log"
Jul 07 14:30:36.302400 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.302369    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:30:36.302854 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.302615    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" containerID="cri-o://586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745" gracePeriod=30
Jul 07 14:30:36.302854 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.302647    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kube-rbac-proxy" containerID="cri-o://2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc" gracePeriod=30
Jul 07 14:30:36.351655 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.351624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:36.358928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.358903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:36.439311 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.439273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:36.489656 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.489637    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/1.log"
Jul 07 14:30:36.491757 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.491743    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:36.569806 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.569734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") pod \"8b6caba9-a942-443b-8dbe-0b462a60d80b\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") "
Jul 07 14:30:36.569806 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.569795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hcbf\" (UniqueName: \"kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf\") pod \"8b6caba9-a942-443b-8dbe-0b462a60d80b\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") "
Jul 07 14:30:36.570067 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.569819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config\") pod \"8b6caba9-a942-443b-8dbe-0b462a60d80b\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") "
Jul 07 14:30:36.570067 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.569842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location\") pod \"8b6caba9-a942-443b-8dbe-0b462a60d80b\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") "
Jul 07 14:30:36.570067 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.569884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert\") pod \"8b6caba9-a942-443b-8dbe-0b462a60d80b\" (UID: \"8b6caba9-a942-443b-8dbe-0b462a60d80b\") "
Jul 07 14:30:36.570245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.570132    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b6caba9-a942-443b-8dbe-0b462a60d80b" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:30:36.570245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.570204    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b6caba9-a942-443b-8dbe-0b462a60d80b" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:36.570345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.570252    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b6caba9-a942-443b-8dbe-0b462a60d80b" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:36.571818 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.571795    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf" (OuterVolumeSpecName: "kube-api-access-8hcbf") pod "8b6caba9-a942-443b-8dbe-0b462a60d80b" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b"). InnerVolumeSpecName "kube-api-access-8hcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:30:36.571952 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.571861    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b6caba9-a942-443b-8dbe-0b462a60d80b" (UID: "8b6caba9-a942-443b-8dbe-0b462a60d80b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:30:36.670772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.670742    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcbf\" (UniqueName: \"kubernetes.io/projected/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-api-access-8hcbf\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:36.670772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.670766    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:36.670772 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.670776    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b6caba9-a942-443b-8dbe-0b462a60d80b-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:36.670986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.670785    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b6caba9-a942-443b-8dbe-0b462a60d80b-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:36.670986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:36.670794    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b6caba9-a942-443b-8dbe-0b462a60d80b-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:37.136066 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.136040    2563 generic.go:356] "Generic (PLEG): container finished" podID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerID="2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc" exitCode=2
Jul 07 14:30:37.136241 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.136133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerDied","Data":"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"}
Jul 07 14:30:37.137185 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.137171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-07975f-predictor-7b8f47b784-gz77k_8b6caba9-a942-443b-8dbe-0b462a60d80b/storage-initializer/1.log"
Jul 07 14:30:37.137252 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.137219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k" event={"ID":"8b6caba9-a942-443b-8dbe-0b462a60d80b","Type":"ContainerDied","Data":"480ebf0caa52b0f509bf5f55bea78bb4f5188059e8b3390184e24c35824e6017"}
Jul 07 14:30:37.137288 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.137256    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"
Jul 07 14:30:37.137345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.137260    2563 scope.go:117] "RemoveContainer" containerID="9bda96941c328bef8548bd1083887ca414710f2bb47a55eb138c8d7e4e57080f"
Jul 07 14:30:37.182290 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.182266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:37.186843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.186817    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-07975f-predictor-7b8f47b784-gz77k"]
Jul 07 14:30:37.734732 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:37.734694    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 14:30:38.087572 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:38.087494    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" path="/var/lib/kubelet/pods/8b6caba9-a942-443b-8dbe-0b462a60d80b/volumes"
Jul 07 14:30:40.083399 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.083358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 14:30:40.845714 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.845690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:30:40.904308 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert\") pod \"37620da4-15da-4d0d-ad8f-e404d3b8e339\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") "
Jul 07 14:30:40.904487 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config\") pod \"37620da4-15da-4d0d-ad8f-e404d3b8e339\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") "
Jul 07 14:30:40.904487 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location\") pod \"37620da4-15da-4d0d-ad8f-e404d3b8e339\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") "
Jul 07 14:30:40.904487 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls\") pod \"37620da4-15da-4d0d-ad8f-e404d3b8e339\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") "
Jul 07 14:30:40.904487 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88dp4\" (UniqueName: \"kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4\") pod \"37620da4-15da-4d0d-ad8f-e404d3b8e339\" (UID: \"37620da4-15da-4d0d-ad8f-e404d3b8e339\") "
Jul 07 14:30:40.904743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904661    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37620da4-15da-4d0d-ad8f-e404d3b8e339" (UID: "37620da4-15da-4d0d-ad8f-e404d3b8e339"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:40.904743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904706    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37620da4-15da-4d0d-ad8f-e404d3b8e339" (UID: "37620da4-15da-4d0d-ad8f-e404d3b8e339"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:30:40.904743 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.904722    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37620da4-15da-4d0d-ad8f-e404d3b8e339" (UID: "37620da4-15da-4d0d-ad8f-e404d3b8e339"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:40.906490 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.906462    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37620da4-15da-4d0d-ad8f-e404d3b8e339" (UID: "37620da4-15da-4d0d-ad8f-e404d3b8e339"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:30:40.906590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:40.906504    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4" (OuterVolumeSpecName: "kube-api-access-88dp4") pod "37620da4-15da-4d0d-ad8f-e404d3b8e339" (UID: "37620da4-15da-4d0d-ad8f-e404d3b8e339"). InnerVolumeSpecName "kube-api-access-88dp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:30:41.005696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.005614    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:41.005696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.005639    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37620da4-15da-4d0d-ad8f-e404d3b8e339-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:41.005696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.005651    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37620da4-15da-4d0d-ad8f-e404d3b8e339-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:41.005696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.005662    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dp4\" (UniqueName: \"kubernetes.io/projected/37620da4-15da-4d0d-ad8f-e404d3b8e339-kube-api-access-88dp4\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:41.005696 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.005670    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37620da4-15da-4d0d-ad8f-e404d3b8e339-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:41.155981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.155942    2563 generic.go:356] "Generic (PLEG): container finished" podID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerID="586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745" exitCode=0
Jul 07 14:30:41.156462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.156019    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"
Jul 07 14:30:41.156462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.156021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerDied","Data":"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"}
Jul 07 14:30:41.156462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.156153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg" event={"ID":"37620da4-15da-4d0d-ad8f-e404d3b8e339","Type":"ContainerDied","Data":"e86fd084a1cf7d4aa6d42388b7080c922142e6aa2aa2cca01a62f1163b062e9c"}
Jul 07 14:30:41.156462 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.156194    2563 scope.go:117] "RemoveContainer" containerID="2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"
Jul 07 14:30:41.164580 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.164564    2563 scope.go:117] "RemoveContainer" containerID="586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"
Jul 07 14:30:41.171716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.171698    2563 scope.go:117] "RemoveContainer" containerID="e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"
Jul 07 14:30:41.178893 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.178870    2563 scope.go:117] "RemoveContainer" containerID="2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"
Jul 07 14:30:41.179181 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:41.179160    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc\": container with ID starting with 2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc not found: ID does not exist" containerID="2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"
Jul 07 14:30:41.179319 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.179190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc"} err="failed to get container status \"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc\": rpc error: code = NotFound desc = could not find container \"2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc\": container with ID starting with 2c7c03fd1c1cc6d9c9851f2dec1662fa1f0f9314a5a40821abd808d71a51c2dc not found: ID does not exist"
Jul 07 14:30:41.179319 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.179213    2563 scope.go:117] "RemoveContainer" containerID="586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"
Jul 07 14:30:41.179652 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:41.179631    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745\": container with ID starting with 586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745 not found: ID does not exist" containerID="586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"
Jul 07 14:30:41.179716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.179659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745"} err="failed to get container status \"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745\": rpc error: code = NotFound desc = could not find container \"586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745\": container with ID starting with 586bd75ea4407c0df942ec7a795fe3f388e5d8e92c0c581429be6c309cb72745 not found: ID does not exist"
Jul 07 14:30:41.179716 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.179675    2563 scope.go:117] "RemoveContainer" containerID="e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"
Jul 07 14:30:41.179933 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:41.179919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3\": container with ID starting with e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3 not found: ID does not exist" containerID="e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"
Jul 07 14:30:41.179981 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.179937    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3"} err="failed to get container status \"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3\": rpc error: code = NotFound desc = could not find container \"e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3\": container with ID starting with e29b3a6a1badcbb1430bf24d7517e7d2869c21b70bf65fd7fe19295beab5fbf3 not found: ID does not exist"
Jul 07 14:30:41.180896 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.180880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:30:41.184785 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.184765    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-07975f-predictor-8574dbdbd7-fvkgg"]
Jul 07 14:30:41.521279 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:41.521572 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521560    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kube-rbac-proxy"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521573    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kube-rbac-proxy"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521582    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="storage-initializer"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521605    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="storage-initializer"
Jul 07 14:30:41.521617 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521616    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kserve-container"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521687    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521695    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" containerName="kube-rbac-proxy"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521742    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.521800 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.521747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6caba9-a942-443b-8dbe-0b462a60d80b" containerName="storage-initializer"
Jul 07 14:30:41.526076 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.526059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.529075 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:30:41.529199 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-da7d24-predictor-serving-cert\""
Jul 07 14:30:41.529199 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:30:41.529199 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529059    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:30:41.529376 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529364    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-da7d24\""
Jul 07 14:30:41.529469 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-da7d24-dockercfg-l6wxt\""
Jul 07 14:30:41.529527 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.529479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-da7d24-kube-rbac-proxy-sar-config\""
Jul 07 14:30:41.530461 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.530441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:41.540288 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.540267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:41.610819 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.610795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.610946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.610828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.610946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.610849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.610946 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.610907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.611049 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.610946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznzf\" (UniqueName: \"kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.711482 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.711450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznzf\" (UniqueName: \"kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.711687 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.711505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.711687 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.711530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.711687 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.711546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.711687 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.711567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.712031 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.712008    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.712296 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.712277    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.712377 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.712343    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.713891 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.713873    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.721591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.721570    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznzf\" (UniqueName: \"kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf\") pod \"isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") " pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.824545 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.824462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:41.836008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.835979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:41.967647 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:41.967617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:41.969115 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:30:41.969069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0b28be_2ecd_4099_be11_6381321e2562.slice/crio-9da0a5f63728b5e5327d387f91e9eca6307922527ac6714565b79f424f6f4de8 WatchSource:0}: Error finding container 9da0a5f63728b5e5327d387f91e9eca6307922527ac6714565b79f424f6f4de8: Status 404 returned error can't find the container with id 9da0a5f63728b5e5327d387f91e9eca6307922527ac6714565b79f424f6f4de8
Jul 07 14:30:42.087683 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:42.087593    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37620da4-15da-4d0d-ad8f-e404d3b8e339" path="/var/lib/kubelet/pods/37620da4-15da-4d0d-ad8f-e404d3b8e339/volumes"
Jul 07 14:30:42.162714 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:42.162676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerStarted","Data":"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"}
Jul 07 14:30:42.163141 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:42.162739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerStarted","Data":"9da0a5f63728b5e5327d387f91e9eca6307922527ac6714565b79f424f6f4de8"}
Jul 07 14:30:47.183537 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:47.183511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/0.log"
Jul 07 14:30:47.184073 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:47.183554    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c0b28be-2ecd-4099-be11-6381321e2562" containerID="a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b" exitCode=1
Jul 07 14:30:47.184073 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:47.183590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerDied","Data":"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"}
Jul 07 14:30:48.190494 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:48.190465    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/0.log"
Jul 07 14:30:48.190877 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:48.190514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerStarted","Data":"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"}
Jul 07 14:30:51.507693 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:51.507617    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:51.508158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:51.507847    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer" containerID="cri-o://43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581" gracePeriod=30
Jul 07 14:30:51.533422 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:51.533396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:53.849932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.849913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/1.log"
Jul 07 14:30:53.850276 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.850261    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/0.log"
Jul 07 14:30:53.852561 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.852547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:53.911534 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert\") pod \"7c0b28be-2ecd-4099-be11-6381321e2562\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") "
Jul 07 14:30:53.911660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config\") pod \"7c0b28be-2ecd-4099-be11-6381321e2562\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") "
Jul 07 14:30:53.911660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911613    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls\") pod \"7c0b28be-2ecd-4099-be11-6381321e2562\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") "
Jul 07 14:30:53.911660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sznzf\" (UniqueName: \"kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf\") pod \"7c0b28be-2ecd-4099-be11-6381321e2562\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") "
Jul 07 14:30:53.911660 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location\") pod \"7c0b28be-2ecd-4099-be11-6381321e2562\" (UID: \"7c0b28be-2ecd-4099-be11-6381321e2562\") "
Jul 07 14:30:53.911907 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911875    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c0b28be-2ecd-4099-be11-6381321e2562" (UID: "7c0b28be-2ecd-4099-be11-6381321e2562"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:53.912006 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911972    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c0b28be-2ecd-4099-be11-6381321e2562" (UID: "7c0b28be-2ecd-4099-be11-6381321e2562"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:30:53.912006 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.911984    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c0b28be-2ecd-4099-be11-6381321e2562" (UID: "7c0b28be-2ecd-4099-be11-6381321e2562"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:30:53.913590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.913565    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c0b28be-2ecd-4099-be11-6381321e2562" (UID: "7c0b28be-2ecd-4099-be11-6381321e2562"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:30:53.913667 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:53.913651    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf" (OuterVolumeSpecName: "kube-api-access-sznzf") pod "7c0b28be-2ecd-4099-be11-6381321e2562" (UID: "7c0b28be-2ecd-4099-be11-6381321e2562"). InnerVolumeSpecName "kube-api-access-sznzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:30:54.013274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.013199    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c0b28be-2ecd-4099-be11-6381321e2562-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:54.013274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.013226    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sznzf\" (UniqueName: \"kubernetes.io/projected/7c0b28be-2ecd-4099-be11-6381321e2562-kube-api-access-sznzf\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:54.013274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.013236    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c0b28be-2ecd-4099-be11-6381321e2562-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:54.013274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.013244    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:54.013274 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.013253    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c0b28be-2ecd-4099-be11-6381321e2562-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:30:54.214786 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.214762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/1.log"
Jul 07 14:30:54.215195 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f_7c0b28be-2ecd-4099-be11-6381321e2562/storage-initializer/0.log"
Jul 07 14:30:54.215308 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215217    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c0b28be-2ecd-4099-be11-6381321e2562" containerID="43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581" exitCode=1
Jul 07 14:30:54.215308 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"
Jul 07 14:30:54.215393 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerDied","Data":"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"}
Jul 07 14:30:54.215393 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f" event={"ID":"7c0b28be-2ecd-4099-be11-6381321e2562","Type":"ContainerDied","Data":"9da0a5f63728b5e5327d387f91e9eca6307922527ac6714565b79f424f6f4de8"}
Jul 07 14:30:54.215475 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.215411    2563 scope.go:117] "RemoveContainer" containerID="43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"
Jul 07 14:30:54.223542 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.223526    2563 scope.go:117] "RemoveContainer" containerID="a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"
Jul 07 14:30:54.230577 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.230555    2563 scope.go:117] "RemoveContainer" containerID="43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"
Jul 07 14:30:54.230810 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:54.230794    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581\": container with ID starting with 43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581 not found: ID does not exist" containerID="43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"
Jul 07 14:30:54.230864 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.230818    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581"} err="failed to get container status \"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581\": rpc error: code = NotFound desc = could not find container \"43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581\": container with ID starting with 43dbec544189ef27453b0afc2f9fd9a2e0fb769cbef849d6d8e442455db8f581 not found: ID does not exist"
Jul 07 14:30:54.230864 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.230832    2563 scope.go:117] "RemoveContainer" containerID="a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"
Jul 07 14:30:54.231070 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:30:54.231053    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b\": container with ID starting with a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b not found: ID does not exist" containerID="a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"
Jul 07 14:30:54.231142 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.231077    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b"} err="failed to get container status \"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b\": rpc error: code = NotFound desc = could not find container \"a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b\": container with ID starting with a39b46de96ccbf71a462dee42b21d306dcc588d0eb045ddcb67331bfe2205f8b not found: ID does not exist"
Jul 07 14:30:54.250087 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.250065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:54.254357 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:54.254336    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da7d24-predictor-65cb8489c4-zxr7f"]
Jul 07 14:30:56.088776 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.088743    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" path="/var/lib/kubelet/pods/7c0b28be-2ecd-4099-be11-6381321e2562/volumes"
Jul 07 14:30:56.689944 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.689911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:56.720177 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720149    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:30:56.720498 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720486    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.720498 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720499    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.720575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720511    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.720575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720517    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.720575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.720575 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.720575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c0b28be-2ecd-4099-be11-6381321e2562" containerName="storage-initializer"
Jul 07 14:30:56.725140 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.725125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.729420 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.729398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:30:56.729420 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.729399    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8rhzg\""
Jul 07 14:30:56.730481 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.730465    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:30:56.730537 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.730480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:30:56.730593 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.730469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-73688-kube-rbac-proxy-sar-config\""
Jul 07 14:30:56.730655 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.730606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:30:56.730945 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.730933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-73688-predictor-serving-cert\""
Jul 07 14:30:56.740879 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.740860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:30:56.835305 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.835277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.835460 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.835322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5lm\" (UniqueName: \"kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.835460 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.835353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.835460 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.835426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.835584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.835460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.922332 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.922303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:30:56.936474 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.936570 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5lm\" (UniqueName: \"kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.936570 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.936570 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.936570 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.937008 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.936984    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.937247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.937223    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.937247 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.937235    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.938931 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.938913    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:56.946251 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:56.946199    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5lm\" (UniqueName: \"kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm\") pod \"raw-sklearn-73688-predictor-65d96b6466-67vtr\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") " pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:57.034896 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:57.034864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:30:57.164160 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:57.164137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:30:57.166631 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:30:57.166601    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0976311d_7f96_496a_8f90_2411fa4978f6.slice/crio-76cb4df30ddc874d53f2bc15549d71b172d3d7040879e5c837e0f8dab38b3b47 WatchSource:0}: Error finding container 76cb4df30ddc874d53f2bc15549d71b172d3d7040879e5c837e0f8dab38b3b47: Status 404 returned error can't find the container with id 76cb4df30ddc874d53f2bc15549d71b172d3d7040879e5c837e0f8dab38b3b47
Jul 07 14:30:57.230055 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:57.230032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerStarted","Data":"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"}
Jul 07 14:30:57.230180 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:30:57.230118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerStarted","Data":"76cb4df30ddc874d53f2bc15549d71b172d3d7040879e5c837e0f8dab38b3b47"}
Jul 07 14:31:01.248444 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:01.248356    2563 generic.go:356] "Generic (PLEG): container finished" podID="0976311d-7f96-496a-8f90-2411fa4978f6" containerID="6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad" exitCode=0
Jul 07 14:31:01.248444 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:01.248404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerDied","Data":"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"}
Jul 07 14:31:02.254781 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:02.254745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerStarted","Data":"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"}
Jul 07 14:31:02.255165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:02.254808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerStarted","Data":"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"}
Jul 07 14:31:02.255165 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:02.254998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:31:02.281624 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:02.281575    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podStartSLOduration=2.637070734 podStartE2EDuration="6.281558173s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644487439s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:31:02.279955505 +0000 UTC m=+1256.836391892" watchObservedRunningTime="2026-07-07 14:31:02.281558173 +0000 UTC m=+1256.837994536"
Jul 07 14:31:03.259339 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:03.259301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:31:03.260557 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:03.260527    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:04.264198 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:04.264157    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:09.268244 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:09.268215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:31:09.268782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:09.268749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:19.269024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:19.268982    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:29.268960 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:29.268912    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:39.269250 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:39.269210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:49.269345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:49.269299    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:31:59.268872 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:31:59.268831    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:32:09.269492 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:09.269452    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:32:13.083724 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:13.083685    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:32:14.384724 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:14.384692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:32:23.084193 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:23.084162    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:32:26.903933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:26.903895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:32:26.904394 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:26.904141    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container" containerID="cri-o://1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693" gracePeriod=30
Jul 07 14:32:26.904394 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:26.904194    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kube-rbac-proxy" containerID="cri-o://ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00" gracePeriod=30
Jul 07 14:32:26.967568 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:26.967540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:32:27.597240 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:27.597210    2563 generic.go:356] "Generic (PLEG): container finished" podID="0976311d-7f96-496a-8f90-2411fa4978f6" containerID="ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00" exitCode=2
Jul 07 14:32:27.597406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:27.597285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerDied","Data":"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"}
Jul 07 14:32:29.265213 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:29.265171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 07 14:32:31.747838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.747813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:32:31.808687 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.808660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw5lm\" (UniqueName: \"kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm\") pod \"0976311d-7f96-496a-8f90-2411fa4978f6\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") "
Jul 07 14:32:31.808838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.808693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location\") pod \"0976311d-7f96-496a-8f90-2411fa4978f6\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") "
Jul 07 14:32:31.808838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.808726    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls\") pod \"0976311d-7f96-496a-8f90-2411fa4978f6\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") "
Jul 07 14:32:31.808838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.808799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config\") pod \"0976311d-7f96-496a-8f90-2411fa4978f6\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") "
Jul 07 14:32:31.808838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.808831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert\") pod \"0976311d-7f96-496a-8f90-2411fa4978f6\" (UID: \"0976311d-7f96-496a-8f90-2411fa4978f6\") "
Jul 07 14:32:31.809166 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.809138    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0976311d-7f96-496a-8f90-2411fa4978f6" (UID: "0976311d-7f96-496a-8f90-2411fa4978f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:32:31.809245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.809184    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0976311d-7f96-496a-8f90-2411fa4978f6" (UID: "0976311d-7f96-496a-8f90-2411fa4978f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:32:31.809245 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.809212    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0976311d-7f96-496a-8f90-2411fa4978f6" (UID: "0976311d-7f96-496a-8f90-2411fa4978f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:32:31.810775 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.810754    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm" (OuterVolumeSpecName: "kube-api-access-vw5lm") pod "0976311d-7f96-496a-8f90-2411fa4978f6" (UID: "0976311d-7f96-496a-8f90-2411fa4978f6"). InnerVolumeSpecName "kube-api-access-vw5lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:32:31.810828 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.810770    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0976311d-7f96-496a-8f90-2411fa4978f6" (UID: "0976311d-7f96-496a-8f90-2411fa4978f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:32:31.910367 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.910337    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5lm\" (UniqueName: \"kubernetes.io/projected/0976311d-7f96-496a-8f90-2411fa4978f6-kube-api-access-vw5lm\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:32:31.910367 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.910362    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0976311d-7f96-496a-8f90-2411fa4978f6-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:32:31.910367 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.910373    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0976311d-7f96-496a-8f90-2411fa4978f6-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:32:31.910584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.910383    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:32:31.910584 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:31.910391    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0976311d-7f96-496a-8f90-2411fa4978f6-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:32:32.058900 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.058867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:32:32.081488 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081461    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:32:32.081797 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081784    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container"
Jul 07 14:32:32.081839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081798    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container"
Jul 07 14:32:32.081839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081805    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kube-rbac-proxy"
Jul 07 14:32:32.081839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081811    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kube-rbac-proxy"
Jul 07 14:32:32.081839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081829    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="storage-initializer"
Jul 07 14:32:32.081839 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081836    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="storage-initializer"
Jul 07 14:32:32.081986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081887    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kserve-container"
Jul 07 14:32:32.081986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.081898    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" containerName="kube-rbac-proxy"
Jul 07 14:32:32.085246 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.085230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.087758 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.087733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-b5c4a-predictor-serving-cert\""
Jul 07 14:32:32.087880 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.087735    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-b5c4a-kube-rbac-proxy-sar-config\""
Jul 07 14:32:32.101714 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.101693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:32:32.112499 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.112474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.112591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.112505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szsb\" (UniqueName: \"kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.112591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.112533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.112671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.112616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.112671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.112657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.213897 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.213810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.213897 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.213866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.213923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.213949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szsb\" (UniqueName: \"kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214139 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.213987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214139 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:32:32.214050    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-serving-cert: secret "raw-sklearn-runtime-b5c4a-predictor-serving-cert" not found
Jul 07 14:32:32.214139 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:32:32.214142    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls podName:a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9 nodeName:}" failed. No retries permitted until 2026-07-07 14:32:32.714121408 +0000 UTC m=+1347.270557754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls") pod "raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9") : secret "raw-sklearn-runtime-b5c4a-predictor-serving-cert" not found
Jul 07 14:32:32.214413 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.214359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214521 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.214504    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.214596 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.214579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.223054 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.223034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szsb\" (UniqueName: \"kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.485346 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.485268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:32:32.618837 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.618805    2563 generic.go:356] "Generic (PLEG): container finished" podID="0976311d-7f96-496a-8f90-2411fa4978f6" containerID="1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693" exitCode=0
Jul 07 14:32:32.619024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.618859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerDied","Data":"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"}
Jul 07 14:32:32.619024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.618869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"
Jul 07 14:32:32.619024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.618909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr" event={"ID":"0976311d-7f96-496a-8f90-2411fa4978f6","Type":"ContainerDied","Data":"76cb4df30ddc874d53f2bc15549d71b172d3d7040879e5c837e0f8dab38b3b47"}
Jul 07 14:32:32.619024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.618936    2563 scope.go:117] "RemoveContainer" containerID="ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"
Jul 07 14:32:32.626984 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.626968    2563 scope.go:117] "RemoveContainer" containerID="1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"
Jul 07 14:32:32.633934 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.633915    2563 scope.go:117] "RemoveContainer" containerID="6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"
Jul 07 14:32:32.638198 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.638170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:32:32.640419 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.640400    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73688-predictor-65d96b6466-67vtr"]
Jul 07 14:32:32.641344 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.641332    2563 scope.go:117] "RemoveContainer" containerID="ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"
Jul 07 14:32:32.641590 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:32:32.641573    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00\": container with ID starting with ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00 not found: ID does not exist" containerID="ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"
Jul 07 14:32:32.641662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.641597    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00"} err="failed to get container status \"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00\": rpc error: code = NotFound desc = could not find container \"ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00\": container with ID starting with ed130fc79d2192c9355e22b2d50c6ddb4bce7ae45d22264a638f642fa7d8fa00 not found: ID does not exist"
Jul 07 14:32:32.641662 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.641615    2563 scope.go:117] "RemoveContainer" containerID="1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"
Jul 07 14:32:32.641827 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:32:32.641809    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693\": container with ID starting with 1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693 not found: ID does not exist" containerID="1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"
Jul 07 14:32:32.641866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.641835    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693"} err="failed to get container status \"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693\": rpc error: code = NotFound desc = could not find container \"1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693\": container with ID starting with 1016ccd6ebc1171da4451a9a29eeffe362b5c394891ff5c5c459f0ef7f204693 not found: ID does not exist"
Jul 07 14:32:32.641866 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.641851    2563 scope.go:117] "RemoveContainer" containerID="6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"
Jul 07 14:32:32.642072 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:32:32.642058    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad\": container with ID starting with 6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad not found: ID does not exist" containerID="6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"
Jul 07 14:32:32.642133 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.642075    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad"} err="failed to get container status \"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad\": rpc error: code = NotFound desc = could not find container \"6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad\": container with ID starting with 6c9298c96de6681de91c20353107de8382ffc8cef04360c6996e260f152dd3ad not found: ID does not exist"
Jul 07 14:32:32.718814 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.718784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.721070 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.721044    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") pod \"raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:32.996388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:32.996352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:33.126425 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:33.126404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:32:33.128515 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:32:33.128492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d6b4ea_71ca_4c11_bb62_79e6c0eb71b9.slice/crio-ff180ac45aa0771dc9aaf6a8387aaf78e5e284ac64a4a9deb42fef17d8c04a85 WatchSource:0}: Error finding container ff180ac45aa0771dc9aaf6a8387aaf78e5e284ac64a4a9deb42fef17d8c04a85: Status 404 returned error can't find the container with id ff180ac45aa0771dc9aaf6a8387aaf78e5e284ac64a4a9deb42fef17d8c04a85
Jul 07 14:32:33.625618 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:33.625577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerStarted","Data":"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"}
Jul 07 14:32:33.625797 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:33.625649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerStarted","Data":"ff180ac45aa0771dc9aaf6a8387aaf78e5e284ac64a4a9deb42fef17d8c04a85"}
Jul 07 14:32:34.087591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:34.087510    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0976311d-7f96-496a-8f90-2411fa4978f6" path="/var/lib/kubelet/pods/0976311d-7f96-496a-8f90-2411fa4978f6/volumes"
Jul 07 14:32:34.202488 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:34.202458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:32:37.643333 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:37.643300    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerID="6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e" exitCode=0
Jul 07 14:32:37.643792 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:37.643354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerDied","Data":"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"}
Jul 07 14:32:38.650243 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.650215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerStarted","Data":"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"}
Jul 07 14:32:38.650596 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.650272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerStarted","Data":"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"}
Jul 07 14:32:38.650596 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.650534    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:38.650708 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.650666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:38.651671 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.651645    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:32:38.672991 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:38.672948    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podStartSLOduration=3.128672736 podStartE2EDuration="6.672935868s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.544263132s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:32:38.671907817 +0000 UTC m=+1353.228344182" watchObservedRunningTime="2026-07-07 14:32:38.672935868 +0000 UTC m=+1353.229372231"
Jul 07 14:32:39.654730 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:39.654695    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:32:44.659933 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:44.659905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:32:44.660446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:44.660421    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:32:54.661339 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:32:54.661298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:04.661163 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:04.661124    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:14.661146 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:14.661085    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:24.660816 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:24.660780    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:34.660370 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:34.660328    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:44.661251 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:44.661206    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:33:54.661731 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:33:54.661703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:34:02.141309 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.141226    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:34:02.141782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.141484    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" containerID="cri-o://ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd" gracePeriod=30
Jul 07 14:34:02.141782 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.141520    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kube-rbac-proxy" containerID="cri-o://d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2" gracePeriod=30
Jul 07 14:34:02.173352 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.173307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:02.986507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.986469    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerID="d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2" exitCode=2
Jul 07 14:34:02.986507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:02.986504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerDied","Data":"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"}
Jul 07 14:34:04.655910 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:04.655867    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 07 14:34:04.660388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:04.660365    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 14:34:06.778922 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.778894    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:34:06.883945 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.883916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") pod \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") "
Jul 07 14:34:06.884116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.883964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert\") pod \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") "
Jul 07 14:34:06.884116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config\") pod \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") "
Jul 07 14:34:06.884116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7szsb\" (UniqueName: \"kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb\") pod \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") "
Jul 07 14:34:06.884116 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location\") pod \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\" (UID: \"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9\") "
Jul 07 14:34:06.884469 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884445    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:34:06.884539 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884440    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:34:06.884539 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.884448    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:34:06.885994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.885974    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:34:06.886128 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.886090    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb" (OuterVolumeSpecName: "kube-api-access-7szsb") pod "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" (UID: "a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9"). InnerVolumeSpecName "kube-api-access-7szsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:34:06.984633 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.984605    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-cabundle-cert\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:06.984633 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.984628    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:06.984633 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.984638    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szsb\" (UniqueName: \"kubernetes.io/projected/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kube-api-access-7szsb\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:06.984841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.984646    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-kserve-provision-location\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:06.984841 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:06.984656    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9-proxy-tls\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:07.004756 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.004722    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerID="ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd" exitCode=0
Jul 07 14:34:07.004885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.004785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerDied","Data":"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"}
Jul 07 14:34:07.004885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.004796    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"
Jul 07 14:34:07.004885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.004859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l" event={"ID":"a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9","Type":"ContainerDied","Data":"ff180ac45aa0771dc9aaf6a8387aaf78e5e284ac64a4a9deb42fef17d8c04a85"}
Jul 07 14:34:07.004885 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.004885    2563 scope.go:117] "RemoveContainer" containerID="d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"
Jul 07 14:34:07.013284 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.013266    2563 scope.go:117] "RemoveContainer" containerID="ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"
Jul 07 14:34:07.020459 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.020440    2563 scope.go:117] "RemoveContainer" containerID="6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"
Jul 07 14:34:07.027720 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.027703    2563 scope.go:117] "RemoveContainer" containerID="d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"
Jul 07 14:34:07.027923 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.027903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:34:07.028018 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:07.028002    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2\": container with ID starting with d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2 not found: ID does not exist" containerID="d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"
Jul 07 14:34:07.028061 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.028026    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2"} err="failed to get container status \"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2\": rpc error: code = NotFound desc = could not find container \"d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2\": container with ID starting with d7df26e81e8bf854296a50db6dee36d01eeafa5ac774ddf0d6fb4791a560d7e2 not found: ID does not exist"
Jul 07 14:34:07.028061 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.028042    2563 scope.go:117] "RemoveContainer" containerID="ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"
Jul 07 14:34:07.028281 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:07.028266    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd\": container with ID starting with ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd not found: ID does not exist" containerID="ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"
Jul 07 14:34:07.028324 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.028287    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd"} err="failed to get container status \"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd\": rpc error: code = NotFound desc = could not find container \"ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd\": container with ID starting with ff082edd014e9ede1dcfa12c403a257ed28a0aadf1e1c7f551a77006041aa4dd not found: ID does not exist"
Jul 07 14:34:07.028324 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.028302    2563 scope.go:117] "RemoveContainer" containerID="6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"
Jul 07 14:34:07.028524 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:07.028507    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e\": container with ID starting with 6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e not found: ID does not exist" containerID="6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"
Jul 07 14:34:07.028562 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.028529    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e"} err="failed to get container status \"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e\": rpc error: code = NotFound desc = could not find container \"6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e\": container with ID starting with 6546a2b08c9dfd8e6c1e62bcd3a229c8e35a135530e6d15af552f2ba0ea2768e not found: ID does not exist"
Jul 07 14:34:07.033192 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:07.033172    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5c4a-predictor-9cf4568fd-wqj8l"]
Jul 07 14:34:08.088040 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.088009    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" path="/var/lib/kubelet/pods/a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9/volumes"
Jul 07 14:34:08.887965 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.887933    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qnqp/must-gather-kjc4c"]
Jul 07 14:34:08.888292 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="storage-initializer"
Jul 07 14:34:08.888345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="storage-initializer"
Jul 07 14:34:08.888345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888304    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container"
Jul 07 14:34:08.888345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888309    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container"
Jul 07 14:34:08.888345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888323    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kube-rbac-proxy"
Jul 07 14:34:08.888345 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kube-rbac-proxy"
Jul 07 14:34:08.888531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888375    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kserve-container"
Jul 07 14:34:08.888531 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.888384    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d6b4ea-71ca-4c11-bb62-79e6c0eb71b9" containerName="kube-rbac-proxy"
Jul 07 14:34:08.891608 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.891594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:08.893932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.893913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4qnqp\"/\"default-dockercfg-l2d5j\""
Jul 07 14:34:08.894071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.894058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qnqp\"/\"openshift-service-ca.crt\""
Jul 07 14:34:08.894267 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.894256    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qnqp\"/\"kube-root-ca.crt\""
Jul 07 14:34:08.900686 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.900665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qnqp/must-gather-kjc4c"]
Jul 07 14:34:08.914260 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.914240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:08.990340 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:08.990318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:09.001464 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.001441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.001535 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.001472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqbt\" (UniqueName: \"kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.102251 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.102169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.102590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.102282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqbt\" (UniqueName: \"kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.102590 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.102429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.114365 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.114344    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqbt\" (UniqueName: \"kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt\") pod \"must-gather-kjc4c\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") " pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.209181 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.209116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:09.334088 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.334018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qnqp/must-gather-kjc4c"]
Jul 07 14:34:09.336227 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:34:09.336200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59380726_c8f8_4b51_8764_3779eff591d7.slice/crio-ccfe07e18f104d4a1e7d865baa4c8996658a2f64a40096448e72d386ea9e12f8 WatchSource:0}: Error finding container ccfe07e18f104d4a1e7d865baa4c8996658a2f64a40096448e72d386ea9e12f8: Status 404 returned error can't find the container with id ccfe07e18f104d4a1e7d865baa4c8996658a2f64a40096448e72d386ea9e12f8
Jul 07 14:34:09.338111 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:09.338084    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:34:10.020507 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:10.020469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" event={"ID":"59380726-c8f8-4b51-8764-3779eff591d7","Type":"ContainerStarted","Data":"ccfe07e18f104d4a1e7d865baa4c8996658a2f64a40096448e72d386ea9e12f8"}
Jul 07 14:34:16.050595 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:16.050563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" event={"ID":"59380726-c8f8-4b51-8764-3779eff591d7","Type":"ContainerStarted","Data":"d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3"}
Jul 07 14:34:16.051058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:16.050606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" event={"ID":"59380726-c8f8-4b51-8764-3779eff591d7","Type":"ContainerStarted","Data":"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"}
Jul 07 14:34:16.074928 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:16.074865    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" podStartSLOduration=2.160743419 podStartE2EDuration="8.074847798s" totalImagesPullingTime="5.914104379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:16.073167699 +0000 UTC m=+1450.629604063" watchObservedRunningTime="2026-07-07 14:34:16.074847798 +0000 UTC m=+1450.631284161"
Jul 07 14:34:38.149002 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:38.148965    2563 generic.go:356] "Generic (PLEG): container finished" podID="59380726-c8f8-4b51-8764-3779eff591d7" containerID="fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07" exitCode=0
Jul 07 14:34:38.149463 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:38.149035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" event={"ID":"59380726-c8f8-4b51-8764-3779eff591d7","Type":"ContainerDied","Data":"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"}
Jul 07 14:34:38.149463 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:38.149346    2563 scope.go:117] "RemoveContainer" containerID="fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"
Jul 07 14:34:38.347468 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:38.347438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qnqp_must-gather-kjc4c_59380726-c8f8-4b51-8764-3779eff591d7/gather/0.log"
Jul 07 14:34:38.805986 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:38.805904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:39.023388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:39.023358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:39.166657 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:39.166633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:42.122833 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:42.122806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vjsqs_565c84fb-b78f-4739-bcd8-216275ac3643/global-pull-secret-syncer/0.log"
Jul 07 14:34:42.243626 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:42.243602    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log"
Jul 07 14:34:42.381982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:42.381915    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-100.ec2.internal_044b16742602656ad17c3b9ae8672de3/haproxy/0.log"
Jul 07 14:34:43.782721 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:43.782692    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4qnqp/must-gather-kjc4c"]
Jul 07 14:34:43.783168 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:43.782868    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-4qnqp/must-gather-kjc4c" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="copy" containerID="cri-o://d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3" gracePeriod=2
Jul 07 14:34:43.785403 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:43.785361    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kjc4c\" is forbidden: User \"system:node:ip-10-0-140-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qnqp\": no relationship found between node 'ip-10-0-140-100.ec2.internal' and this object" podUID="59380726-c8f8-4b51-8764-3779eff591d7" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:43.785623 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:43.785604    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4qnqp/must-gather-kjc4c"]
Jul 07 14:34:44.013215 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.013193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qnqp_must-gather-kjc4c_59380726-c8f8-4b51-8764-3779eff591d7/copy/0.log"
Jul 07 14:34:44.014604 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.014585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:44.016959 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:44.016935    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kjc4c\" is forbidden: User \"system:node:ip-10-0-140-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qnqp\": no relationship found between node 'ip-10-0-140-100.ec2.internal' and this object" podUID="59380726-c8f8-4b51-8764-3779eff591d7" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:44.101932 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.101915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gqbt\" (UniqueName: \"kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt\") pod \"59380726-c8f8-4b51-8764-3779eff591d7\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") "
Jul 07 14:34:44.102053 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.101949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output\") pod \"59380726-c8f8-4b51-8764-3779eff591d7\" (UID: \"59380726-c8f8-4b51-8764-3779eff591d7\") "
Jul 07 14:34:44.103385 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.103363    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "59380726-c8f8-4b51-8764-3779eff591d7" (UID: "59380726-c8f8-4b51-8764-3779eff591d7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:34:44.104059 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.104038    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt" (OuterVolumeSpecName: "kube-api-access-5gqbt") pod "59380726-c8f8-4b51-8764-3779eff591d7" (UID: "59380726-c8f8-4b51-8764-3779eff591d7"). InnerVolumeSpecName "kube-api-access-5gqbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:34:44.173601 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.173580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qnqp_must-gather-kjc4c_59380726-c8f8-4b51-8764-3779eff591d7/copy/0.log"
Jul 07 14:34:44.173930 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.173906    2563 generic.go:356] "Generic (PLEG): container finished" podID="59380726-c8f8-4b51-8764-3779eff591d7" containerID="d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3" exitCode=143
Jul 07 14:34:44.174044 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.173958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qnqp/must-gather-kjc4c"
Jul 07 14:34:44.174044 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.173992    2563 scope.go:117] "RemoveContainer" containerID="d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3"
Jul 07 14:34:44.181548 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.181533    2563 scope.go:117] "RemoveContainer" containerID="fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"
Jul 07 14:34:44.194074 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.194057    2563 scope.go:117] "RemoveContainer" containerID="d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3"
Jul 07 14:34:44.194351 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:44.194335    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3\": container with ID starting with d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3 not found: ID does not exist" containerID="d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3"
Jul 07 14:34:44.194401 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.194359    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3"} err="failed to get container status \"d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3\": rpc error: code = NotFound desc = could not find container \"d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3\": container with ID starting with d4eacbe75bf2ff8d55d3f04b6f9514c24b32668fb735de129d5fed2f6cf2aef3 not found: ID does not exist"
Jul 07 14:34:44.194401 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.194379    2563 scope.go:117] "RemoveContainer" containerID="fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"
Jul 07 14:34:44.194619 ip-10-0-140-100 kubenswrapper[2563]: E0707 14:34:44.194603    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07\": container with ID starting with fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07 not found: ID does not exist" containerID="fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"
Jul 07 14:34:44.194664 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.194625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07"} err="failed to get container status \"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07\": rpc error: code = NotFound desc = could not find container \"fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07\": container with ID starting with fdeedca4c514de047c3dc931682d05206e0c482892fcb062c2ecae6c3d2aba07 not found: ID does not exist"
Jul 07 14:34:44.202982 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.202963    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gqbt\" (UniqueName: \"kubernetes.io/projected/59380726-c8f8-4b51-8764-3779eff591d7-kube-api-access-5gqbt\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:44.203058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:44.202984    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59380726-c8f8-4b51-8764-3779eff591d7-must-gather-output\") on node \"ip-10-0-140-100.ec2.internal\" DevicePath \"\""
Jul 07 14:34:45.208446 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:45.208415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:45.299564 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:45.299535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkxrc_9fd4c208-30a8-4b72-b694-87fcfd85d46f/konnectivity-agent/0.log" limit=1024
Jul 07 14:34:46.087750 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:46.087713    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59380726-c8f8-4b51-8764-3779eff591d7" path="/var/lib/kubelet/pods/59380726-c8f8-4b51-8764-3779eff591d7/volumes"
Jul 07 14:34:46.864925 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:46.864890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jzd5g_e4f5d889-55eb-452a-919f-6fa70cbd4918/kube-state-metrics/0.log"
Jul 07 14:34:46.892362 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:46.892339    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jzd5g_e4f5d889-55eb-452a-919f-6fa70cbd4918/kube-rbac-proxy-main/0.log"
Jul 07 14:34:46.931397 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:46.931366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jzd5g_e4f5d889-55eb-452a-919f-6fa70cbd4918/kube-rbac-proxy-self/0.log"
Jul 07 14:34:47.454030 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.454005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99jz_2fd5d7c9-f3b7-4ead-8057-d666debb1c90/node-exporter/0.log"
Jul 07 14:34:47.499203 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.499175    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99jz_2fd5d7c9-f3b7-4ead-8057-d666debb1c90/kube-rbac-proxy/0.log"
Jul 07 14:34:47.536922 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.536899    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99jz_2fd5d7c9-f3b7-4ead-8057-d666debb1c90/init-textfile/0.log"
Jul 07 14:34:47.581571 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.581507    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhrvm_cd358bc3-d7d1-4c29-9c87-38fe1e10af7a/kube-rbac-proxy-main/0.log"
Jul 07 14:34:47.616748 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.616728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhrvm_cd358bc3-d7d1-4c29-9c87-38fe1e10af7a/kube-rbac-proxy-self/0.log"
Jul 07 14:34:47.646925 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.646906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mhrvm_cd358bc3-d7d1-4c29-9c87-38fe1e10af7a/openshift-state-metrics/0.log"
Jul 07 14:34:47.702420 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.702395    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/prometheus/0.log"
Jul 07 14:34:47.731304 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.731279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/config-reloader/0.log"
Jul 07 14:34:47.759904 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.759878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/thanos-sidecar/0.log"
Jul 07 14:34:47.789861 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.789828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/kube-rbac-proxy-web/0.log"
Jul 07 14:34:47.827307 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.827275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/kube-rbac-proxy/0.log"
Jul 07 14:34:47.859919 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.859900    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/kube-rbac-proxy-thanos/0.log"
Jul 07 14:34:47.889038 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.889015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_026918b7-21b5-47f0-b8b8-125ab13d6c77/init-config-reloader/0.log"
Jul 07 14:34:47.938512 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.938490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-ntdxf_ae7cb200-27a5-4a05-9e4f-5879de435147/prometheus-operator/0.log"
Jul 07 14:34:47.979254 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:47.979233    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-ntdxf_ae7cb200-27a5-4a05-9e4f-5879de435147/kube-rbac-proxy/0.log"
Jul 07 14:34:48.026024 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.026000    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-5r6p7_d4f94153-6931-4323-b818-3820cac7ea59/prometheus-operator-admission-webhook/0.log"
Jul 07 14:34:48.174516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.174419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/thanos-query/0.log"
Jul 07 14:34:48.212629 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.212608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/kube-rbac-proxy-web/0.log"
Jul 07 14:34:48.242089 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.242063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/kube-rbac-proxy/0.log"
Jul 07 14:34:48.269363 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.269336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/prom-label-proxy/0.log"
Jul 07 14:34:48.300606 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.300585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/kube-rbac-proxy-rules/0.log"
Jul 07 14:34:48.326838 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:48.326817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85744b65d9-86mjq_2373bf6b-da4b-43bd-a9d5-abebf9473971/kube-rbac-proxy-metrics/0.log"
Jul 07 14:34:49.993870 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.993832    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"]
Jul 07 14:34:49.994271 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994171    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="copy"
Jul 07 14:34:49.994271 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994184    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="copy"
Jul 07 14:34:49.994271 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994210    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="gather"
Jul 07 14:34:49.994271 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="gather"
Jul 07 14:34:49.994406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994275    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="gather"
Jul 07 14:34:49.994406 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.994283    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59380726-c8f8-4b51-8764-3779eff591d7" containerName="copy"
Jul 07 14:34:49.998425 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:49.998404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.001409 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.001381    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x5hr6\"/\"default-dockercfg-kkxpj\""
Jul 07 14:34:50.001534 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.001517    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x5hr6\"/\"openshift-service-ca.crt\""
Jul 07 14:34:50.002491 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.002473    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x5hr6\"/\"kube-root-ca.crt\""
Jul 07 14:34:50.013643 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.013624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"]
Jul 07 14:34:50.150890 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.150865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-podres\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.150890 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.150892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668c2\" (UniqueName: \"kubernetes.io/projected/7efeed61-ac2f-4efa-96ec-72d98eb79237-kube-api-access-668c2\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.151071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.150928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-sys\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.151071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.150997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-lib-modules\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.151071 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.151019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-proc\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.251859 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.251781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-podres\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.251859 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.251822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668c2\" (UniqueName: \"kubernetes.io/projected/7efeed61-ac2f-4efa-96ec-72d98eb79237-kube-api-access-668c2\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.251880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-sys\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.251908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-lib-modules\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.251962    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-podres\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.252002    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-sys\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.252007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-lib-modules\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252058 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.252036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-proc\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.252306 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.252133    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7efeed61-ac2f-4efa-96ec-72d98eb79237-proc\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.261452 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.261434    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668c2\" (UniqueName: \"kubernetes.io/projected/7efeed61-ac2f-4efa-96ec-72d98eb79237-kube-api-access-668c2\") pod \"perf-node-gather-daemonset-sz5tn\" (UID: \"7efeed61-ac2f-4efa-96ec-72d98eb79237\") " pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.308210 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.308192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:50.434416 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.434391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"]
Jul 07 14:34:50.436706 ip-10-0-140-100 kubenswrapper[2563]: W0707 14:34:50.436680    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7efeed61_ac2f_4efa_96ec_72d98eb79237.slice/crio-a8b8ee63cfdb4dee51e3ac12ad6b9de9fd6ba659619ba7d2143c61fbf6d57eba WatchSource:0}: Error finding container a8b8ee63cfdb4dee51e3ac12ad6b9de9fd6ba659619ba7d2143c61fbf6d57eba: Status 404 returned error can't find the container with id a8b8ee63cfdb4dee51e3ac12ad6b9de9fd6ba659619ba7d2143c61fbf6d57eba
Jul 07 14:34:50.590085 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:50.590061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-22hzd_5c9152bf-f945-4dab-9072-9fd61b940531/download-server/0.log"
Jul 07 14:34:51.201969 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:51.201931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn" event={"ID":"7efeed61-ac2f-4efa-96ec-72d98eb79237","Type":"ContainerStarted","Data":"8739f7c87f91851a781976897359a71a0737b85caf7476f9daf0a623cb101b35"}
Jul 07 14:34:51.202388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:51.202025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn" event={"ID":"7efeed61-ac2f-4efa-96ec-72d98eb79237","Type":"ContainerStarted","Data":"a8b8ee63cfdb4dee51e3ac12ad6b9de9fd6ba659619ba7d2143c61fbf6d57eba"}
Jul 07 14:34:51.202388 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:51.202055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:51.227843 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:51.227792    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn" podStartSLOduration=2.227777177 podStartE2EDuration="2.227777177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:51.225410226 +0000 UTC m=+1485.781846604" watchObservedRunningTime="2026-07-07 14:34:51.227777177 +0000 UTC m=+1485.784213541"
Jul 07 14:34:52.200018 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:52.199977    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k87n5_dc389be8-7df3-4b36-abc0-68784e3a6f62/dns/0.log"
Jul 07 14:34:52.231773 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:52.231749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k87n5_dc389be8-7df3-4b36-abc0-68784e3a6f62/kube-rbac-proxy/0.log"
Jul 07 14:34:52.416829 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:52.416803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h5x7b_cbad1b4d-6888-4aeb-a6bb-7cdcf59f8e5b/dns-node-resolver/0.log"
Jul 07 14:34:53.005374 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:53.005348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59ff86cfdf-clb4h_a70b3d92-1a64-473c-8aff-a78188c1050b/registry/0.log"
Jul 07 14:34:53.088978 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:53.088952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k8qgz_c85353ad-cb20-4ae6-8bf0-36406ff4c93e/node-ca/0.log"
Jul 07 14:34:54.450027 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:54.449998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ss2mx_c1320315-4957-4380-bede-1ceaef9623d7/serve-healthcheck-canary/0.log"
Jul 07 14:34:55.005430 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:55.005403    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4hzs_58f834c2-b8a4-4d60-8ee9-4b4294accac9/kube-rbac-proxy/0.log"
Jul 07 14:34:55.027364 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:55.027343    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4hzs_58f834c2-b8a4-4d60-8ee9-4b4294accac9/exporter/0.log"
Jul 07 14:34:55.054308 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:55.054286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4hzs_58f834c2-b8a4-4d60-8ee9-4b4294accac9/extractor/0.log"
Jul 07 14:34:57.215170 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:57.215143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x5hr6/perf-node-gather-daemonset-sz5tn"
Jul 07 14:34:57.547815 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:57.547735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-68nb5_709c4649-a4c5-48bf-9d61-7a5f3851e937/s3-init/0.log"
Jul 07 14:34:57.564516 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:57.564499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-68nb5_709c4649-a4c5-48bf-9d61-7a5f3851e937/download-hf-model/0.log"
Jul 07 14:34:57.594243 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:34:57.594221    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-scbn4_ce81d7c8-9e7b-4297-9257-0bfacf9fd5ec/seaweedfs/0.log"
Jul 07 14:35:02.307718 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:02.307688    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-fscgf_f4d69376-31a3-408b-be87-2f780c540cbd/migrator/0.log"
Jul 07 14:35:02.332518 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:02.332485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-fscgf_f4d69376-31a3-408b-be87-2f780c540cbd/graceful-termination/0.log"
Jul 07 14:35:04.575203 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.575173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/kube-multus-additional-cni-plugins/0.log"
Jul 07 14:35:04.635670 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.635642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/egress-router-binary-copy/0.log"
Jul 07 14:35:04.689817 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.689792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/cni-plugins/0.log"
Jul 07 14:35:04.781321 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.781292    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/bond-cni-plugin/0.log"
Jul 07 14:35:04.821771 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.821744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/routeoverride-cni/0.log"
Jul 07 14:35:04.866765 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.866738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/whereabouts-cni-bincopy/0.log"
Jul 07 14:35:04.900486 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.900458    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xk4q9_6cc543ad-56d9-474e-aac9-21e6f2e00224/whereabouts-cni/0.log"
Jul 07 14:35:04.985779 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:04.985743    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z7mkx_bc634bdd-0d3d-4b71-90f5-5449569cac27/kube-multus/0.log"
Jul 07 14:35:05.181591 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:05.181516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hp9vf_00f7d4fa-1d46-4d30-964e-ab7dac77df46/network-metrics-daemon/0.log"
Jul 07 14:35:05.225739 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:05.225709    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hp9vf_00f7d4fa-1d46-4d30-964e-ab7dac77df46/kube-rbac-proxy/0.log"
Jul 07 14:35:06.429293 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.429260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/ovn-controller/0.log"
Jul 07 14:35:06.513087 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.513059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/ovn-acl-logging/0.log"
Jul 07 14:35:06.557167 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.557144    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/kube-rbac-proxy-node/0.log"
Jul 07 14:35:06.608582 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.608555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 14:35:06.671994 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.671973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/northd/0.log"
Jul 07 14:35:06.723072 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.723006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/nbdb/0.log"
Jul 07 14:35:06.769745 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.769718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/sbdb/0.log"
Jul 07 14:35:06.841158 ip-10-0-140-100 kubenswrapper[2563]: I0707 14:35:06.841128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dnd7_487a3a9c-267e-4ac8-82c5-00a1de9a0d9c/ovnkube-controller/0.log"