Jul 01 20:46:48.961993 ip-10-0-135-188 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 20:46:49.382804 ip-10-0-135-188 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:49.382804 ip-10-0-135-188 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 20:46:49.382804 ip-10-0-135-188 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:49.382804 ip-10-0-135-188 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 20:46:49.382804 ip-10-0-135-188 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:49.385250 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.385168    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 20:46:49.390044 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390029    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.390044 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390045    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390050    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390053    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390056    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390059    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390062    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390065    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390067    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390070    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390074    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390078    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390082    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390085    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390088    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390090    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390093    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390095    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390099    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390103    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.390107 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390106    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390109    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390113    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390116    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390119    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390122    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390124    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390127    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390131    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390133    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390136    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390138    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390141    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390143    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390146    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390148    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390151    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390153    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390156    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390158    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.390592 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390161    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390163    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390166    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390168    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390174    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390176    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390179    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390181    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390184    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390186    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390189    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390191    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390193    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390196    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390199    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390202    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390206    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390209    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390212    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390215    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.391337 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390217    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390220    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390223    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390225    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390228    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390231    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390233    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390236    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390238    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390240    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390243    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390245    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390248    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390250    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390252    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390255    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390259    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390262    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390264    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.391850 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390267    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390270    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390273    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390275    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390278    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390280    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390283    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390286    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390288    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390291    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390293    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390296    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390298    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390300    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390303    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390305    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390308    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390310    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390312    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.392305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.390315    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.395516 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395499    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.395516 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395516    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395520    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395524    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395526    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395530    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395533    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395535    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395538    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395540    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395543    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395546    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395548    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395551    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395553    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395556    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395559    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395561    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395564    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395566    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395569    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.395588 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395571    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395574    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395576    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395579    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395581    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395585    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395590    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395593    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395596    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395598    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395601    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395604    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395607    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395609    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395612    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395614    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395616    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395619    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395621    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395624    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.396084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395626    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395628    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395631    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395633    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395636    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395639    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395641    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395644    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395646    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395649    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395651    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395653    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395656    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395658    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395661    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395664    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395667    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395670    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395672    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.396566 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395676    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395681    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395684    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395686    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395689    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395692    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395694    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395697    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395699    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395702    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395705    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395708    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395729    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395734    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395738    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395742    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395744    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395747    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395751    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395754    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.397043 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395757    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395760    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395763    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395767    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395769    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395772    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395774    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395778    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395781    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395784    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395786    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395789    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395792    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395794    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395796    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395799    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395801    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395804    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.395806    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396189    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.397536 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396195    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396198    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396200    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396203    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396206    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396208    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396211    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396213    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396216    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396218    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396221    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396223    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396226    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396228    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396231    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396234    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396236    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396239    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396241    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396244    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396247    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.398039 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396249    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396252    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396256    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396260    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396262    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396265    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396267    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396270    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396272    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396274    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396277    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396280    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396283    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396285    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396287    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396290    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396292    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396294    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396297    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396299    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.398535 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396302    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396304    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396307    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396309    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396312    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396314    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396316    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396319    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396322    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396324    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396327    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396329    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396331    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396334    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396336    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396339    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396341    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396343    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396346    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.399052 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396369    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396373    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396377    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396381    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396383    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396386    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396389    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396392    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396394    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396397    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396399    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396401    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396404    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396406    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396409    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396412    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396414    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396417    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396420    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.399507 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396422    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396425    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396427    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396430    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396433    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396436    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396438    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396442    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396444    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396446    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396449    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396451    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396454    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396456    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396459    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396461    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396464    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396466    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.396470    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.400006 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396534    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396541    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396547    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396552    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396561    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396564    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396568    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396573    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396577    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396580    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396583    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396586    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396589    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396592    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396594    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396597    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396601    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396604    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396608    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396611    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396614    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396617    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396620    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396624    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 20:46:49.400450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396627    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396629    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396632    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396635    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396638    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396641    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396644    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396646    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396650    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396652    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396655    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396658    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396660    2574 flags.go:64] FLAG: --enable-server="true"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396663    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396667    2574 flags.go:64] FLAG: --event-burst="100"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396670    2574 flags.go:64] FLAG: --event-qps="50"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396673    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396676    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396680    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396683    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396686    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396689    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396692    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396694    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396698    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 20:46:49.401035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396702    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396704    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396707    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396724    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396727    2574 flags.go:64] FLAG: --feature-gates=""
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396730    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396733    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396736    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396739    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396742    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396745    2574 flags.go:64] FLAG: --help="false"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396748    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396751    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396754    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396756    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396759    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396763    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396765    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396768    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396771    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396774    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396777    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396780    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396782    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 01 20:46:49.401626 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396785    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396788    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396790    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396793    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396796    2574 flags.go:64] FLAG: --lock-file=""
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396798    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396801    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396804    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396810    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396813    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396815    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396818    2574 flags.go:64] FLAG: --logging-format="text"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396821    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396824    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396826    2574 flags.go:64] FLAG: --manifest-url=""
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396829    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396833    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396836    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396840    2574 flags.go:64] FLAG: --max-pods="110"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396843    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396846    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396848    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396851    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396854    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396857    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 20:46:49.402224 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396859    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396866    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396869    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396872    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396875    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396877    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396883    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396886    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396889    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396891    2574 flags.go:64] FLAG: --port="10250"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396894    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396897    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ef973e497154d607"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396899    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396902    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396906    2574 flags.go:64] FLAG: --register-node="true"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396908    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396911    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396915    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396918    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396921    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396924    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396927    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396930    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396933    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396935    2574 flags.go:64] FLAG: --runonce="false"
Jul 01 20:46:49.402950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396938    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396941    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396943    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396946    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396949    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396952    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396954    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396957    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396960    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396963    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396967    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396970    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396973    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396976    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396979    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396984    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396986    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396989    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396993    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396996    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.396998    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397001    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397004    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397017    2574 flags.go:64] FLAG: --v="2"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397021    2574 flags.go:64] FLAG: --version="false"
Jul 01 20:46:49.403691 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397026    2574 flags.go:64] FLAG: --vmodule=""
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397029    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.397033    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397147    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397151    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397155    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397157    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397160    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397162    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397165    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397167    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397170    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397173    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397175    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397178    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397181    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397183    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397187    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397190    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397192    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.404764 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397195    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397197    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397200    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397202    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397205    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397207    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397210    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397212    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397215    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397217    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397220    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397222    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397224    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397227    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397230    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397232    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397234    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397237    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397239    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.405634 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397242    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397244    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397246    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397249    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397251    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397254    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397256    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397259    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397261    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397263    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397267    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397269    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397272    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397274    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397277    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397279    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397281    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397284    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397286    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.406452 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397288    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397291    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397293    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397296    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397299    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397301    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397303    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397306    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397309    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397311    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397314    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397316    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397319    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397321    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397324    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397326    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397329    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397331    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397333    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397336    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.407305 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397338    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397341    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397344    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397351    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397353    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397356    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397358    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397361    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397363    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397365    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397368    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397371    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397374    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397380    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397383    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397386    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397388    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397391    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397393    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.407968 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397396    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397398    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397401    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397404    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.397406    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.398095    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.406084    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.406105    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.406129    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.406138    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406243    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406251    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406256    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406262    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406266    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.408653 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406271    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406275    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406280    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406284    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406289    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406293    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406298    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406303    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406307    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406311    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406334    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406341    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406346    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406351    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406356    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406361    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406365    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406369    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406374    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406380    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.409382 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406385    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406389    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406393    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406397    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406402    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406407    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406411    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406416    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406420    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406424    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406428    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406432    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406436    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406440    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406445    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406449    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406453    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406456    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406461    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406465    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.410056 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406469    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406473    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406477    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406481    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406486    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406490    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406495    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406501    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406506    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406510    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406515    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406519    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406525    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406529    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406533    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406537    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406542    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406549    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406554    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.410661 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406559    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406563    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406568    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406572    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406577    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406581    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406586    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406590    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406595    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406598    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406603    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406607    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406611    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406616    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406620    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406625    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406629    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406633    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406637    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406641    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.411168 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406645    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406649    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406653    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406657    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406662    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406666    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406671    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406675    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406680    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406685    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406689    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406693    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406697    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406703    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.411648 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406708    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.406747    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406962    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406970    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406975    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406979    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406984    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406988    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406992    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.406997    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407001    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407005    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407009    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407014    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407018    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:49.412022 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407022    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407026    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407031    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407035    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407039    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407043    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407047    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407051    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407056    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407060    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407065    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407070    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407074    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407079    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407084    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407087    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407092    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407098    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407104    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:49.412432 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407109    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407115    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407119    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407124    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407128    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407133    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407138    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407142    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407146    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407150    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407154    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407158    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407162    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407166    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407170    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407174    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407178    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407183    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407187    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407191    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:49.412905 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407195    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407199    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407204    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407208    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407213    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407218    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407222    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407226    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407230    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407234    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407238    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407242    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407246    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407250    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407254    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407261    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407266    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407270    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407275    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:49.413457 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407279    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407283    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407287    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407291    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407295    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407299    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407303    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407307    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407313    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407318    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407322    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407326    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407330    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407334    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407338    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407342    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407347    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407352    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407357    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407361    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:49.413938 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407365    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407369    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407373    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407377    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407382    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407386    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407390    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:49.407394    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.407402    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.407548    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.410225    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.412289    2574 server.go:1019] "Starting client certificate rotation"
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.412393    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:49.414423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.412424    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:49.415990 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.415967    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:49.417276 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.417257    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:49.426745 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.426728    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 01 20:46:49.430065 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.430049    2574 log.go:25] "Validated CRI v1 image API"
Jul 01 20:46:49.430910 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.430894    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 20:46:49.433538 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.433520    2574 fs.go:135] Filesystem UUIDs: map[35e548df-2847-487f-af7d-fe35f1ac1e28:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89765b96-08cc-47aa-8515-9d82ffc9cf31:/dev/nvme0n1p4]
Jul 01 20:46:49.433592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.433539    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 20:46:49.440497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.440391    2574 manager.go:216] Machine: {Timestamp:2026-07-01 20:46:49.43782064 +0000 UTC m=+0.353098669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2996859 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a19dd1f0e6e32fcd4baa2dbb4a0c8 SystemUUID:ec2a19dd-1f0e-6e32-fcd4-baa2dbb4a0c8 BootID:3fef3e49-195e-40cf-8adb-4a78fd796db7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:84:81:dd:1d:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:84:81:dd:1d:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:3a:80:d2:74:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 20:46:49.440497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.440496    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 20:46:49.440599 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.440567    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 20:46:49.443750 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.443724    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 20:46:49.443883 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.443752    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 20:46:49.445427 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.445416    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 20:46:49.445466 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.445430    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 20:46:49.445466 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.445438    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:49.446182 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.446173    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:49.447051 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.447036    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:49.447051 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.447050    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 20:46:49.450304 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450287    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:49.450404 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450395    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 20:46:49.450433 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450426    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 20:46:49.450463 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450446    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 20:46:49.450493 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450465    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 20:46:49.450493 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450478    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 01 20:46:49.450493 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450491    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 20:46:49.450933 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450920    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:49.450994 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.450940    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:49.453340 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.453323    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 20:46:49.455236 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.455213    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 20:46:49.455298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.455248    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 20:46:49.456091 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456077    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 20:46:49.456160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456100    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 20:46:49.456160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456110    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 20:46:49.456160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456119    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 20:46:49.456160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456134    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 20:46:49.456160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456143    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 20:46:49.456538 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456528    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 20:46:49.456538 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456538    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 20:46:49.456592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456549    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 20:46:49.456592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456555    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 20:46:49.456592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456564    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 20:46:49.456592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.456571    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 20:46:49.457481 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.457470    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 20:46:49.457515 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.457483    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 20:46:49.460682 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.460667    2574 server.go:1268] "Started kubelet"
Jul 01 20:46:49.460853 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.460763    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 20:46:49.460932 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.460885    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 20:46:49.460983 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.460954    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 20:46:49.461557 ip-10-0-135-188 systemd[1]: Started Kubernetes Kubelet.
Jul 01 20:46:49.464286 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.464069    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 20:46:49.465342 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.465312    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 20:46:49.465430 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.465329    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 20:46:49.465547 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.465526    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 20:46:49.465975 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.465958    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 20:46:49.467110 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.467087    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 20:46:49.470175 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.469096    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18be45669a28bc05  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-01 20:46:49.460636677 +0000 UTC m=+0.375914714,LastTimestamp:2026-07-01 20:46:49.460636677 +0000 UTC m=+0.375914714,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 01 20:46:49.472862 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.472849    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 20:46:49.472934 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.472872    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:49.472988 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.472932    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.474684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.473739    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 20:46:49.474684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.473757    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 20:46:49.474684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.473908    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 20:46:49.474684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.473920    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 20:46:49.474940 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.474691    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 20:46:49.475654 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.475633    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 20:46:49.476114 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.475940    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 20:46:49.476380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476363    2574 factory.go:55] Registering systemd factory
Jul 01 20:46:49.476453 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476385    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 01 20:46:49.476749 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476731    2574 factory.go:153] Registering CRI-O factory
Jul 01 20:46:49.476749 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476751    2574 factory.go:223] Registration of the crio container factory successfully
Jul 01 20:46:49.476854 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476795    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 20:46:49.476854 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476814    2574 factory.go:103] Registering Raw factory
Jul 01 20:46:49.476854 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.476825    2574 manager.go:1196] Started watching for new ooms in manager
Jul 01 20:46:49.477114 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.477103    2574 manager.go:349] Starting recovery of all containers
Jul 01 20:46:49.488073 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.488057    2574 manager.go:354] Recovery completed
Jul 01 20:46:49.492199 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.492187    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.493923 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.493907    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.493981 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.493936    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.493981 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.493945    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.494403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.494388    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 20:46:49.494455 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.494404    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 20:46:49.494455 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.494422    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:49.494517 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.494391    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.495138 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495123    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.495215 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495156    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.495215 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495172    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.495727 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495700    2574 policy_none.go:49] "None policy: Start"
Jul 01 20:46:49.495787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495735    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 20:46:49.495787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.495746    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 20:46:49.496909 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.496898    2574 policy_none.go:47] "Start"
Jul 01 20:46:49.503918 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.503891    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5blv"
Jul 01 20:46:49.505796 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.505729    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18be45669c24a509  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-188.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-01 20:46:49.493923081 +0000 UTC m=+0.409201109,LastTimestamp:2026-07-01 20:46:49.493923081 +0000 UTC m=+0.409201109,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 01 20:46:49.512190 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.512172    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5blv"
Jul 01 20:46:49.517956 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.517868    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18be45669c24e853  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-188.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-01 20:46:49.493940307 +0000 UTC m=+0.409218335,LastTimestamp:2026-07-01 20:46:49.493940307 +0000 UTC m=+0.409218335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 01 20:46:49.535315 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535291    2574 manager.go:337] "Starting Device Plugin manager"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.535330    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535339    2574 server.go:87] "Starting device plugin registration server"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535588    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535598    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535676    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535777    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.535785    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.536274    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.536334    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.541281    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.542699    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.542762    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.542786    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.542824    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 20:46:49.547491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.545139    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:49.636121 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.636030    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.637081 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.637061    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.637081 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.637093    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.637257 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.637108    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.637257 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.637135    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.643243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.643223    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"]
Jul 01 20:46:49.643310 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.643292    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.644163 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.644148    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.644247 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.644174    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.644247 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.644185    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.645690 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.645677    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.645820 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.645801    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.645860 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.645830    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-188.ec2.internal\": node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.645860 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.645848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.645934 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.645875    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.646401 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646384    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.646486 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646423    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.646486 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646439    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.646555 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646493    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.646555 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646512    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.646555 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.646521    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.647835 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.647820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.647895 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.647847    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:49.648502 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.648487    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:49.648583 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.648516    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:49.648583 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.648528    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:49.672268 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.672248    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-188.ec2.internal\" not found" node="ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.674943 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.674917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.676821 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.676802    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-188.ec2.internal\" not found" node="ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.684148 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.684131    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.775577 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.775542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.775577 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.775575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.775801 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.775593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e0ed2ae21cf9ca9826a478a311bf2a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"a5e0ed2ae21cf9ca9826a478a311bf2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.775801 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.775614    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.784801 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.784782    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.875880 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.875847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.876008 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.875881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e0ed2ae21cf9ca9826a478a311bf2a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"a5e0ed2ae21cf9ca9826a478a311bf2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.876008 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.875930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e0ed2ae21cf9ca9826a478a311bf2a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"a5e0ed2ae21cf9ca9826a478a311bf2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.876008 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.875947    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9c4813851e778afe9f5a0f08ee189fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"a9c4813851e778afe9f5a0f08ee189fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.884981 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.884961    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:49.975550 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.975491    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.980856 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:49.980732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:49.985775 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:49.985755    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:50.086320 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:50.086296    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:50.187022 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:50.187002    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:50.287610 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:50.287545    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:50.388010 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:50.387984    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 01 20:46:50.411732 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.411680    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 20:46:50.411897 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.411873    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 20:46:50.476415 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.476386    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:50.516580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.516542    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 20:41:49 +0000 UTC" deadline="2027-12-15 01:59:10.27212397 +0000 UTC"
Jul 01 20:46:50.516580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.516568    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12749h12m19.75555826s"
Jul 01 20:46:50.573397 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.573328    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:50.586294 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.586272    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:50.588378 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.588362    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 01 20:46:50.592517 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.592500    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:50.600681 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.600665    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:50.705928 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:50.705890    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0ed2ae21cf9ca9826a478a311bf2a.slice/crio-139b1cf65028018c0794af9130c90de3a34cecb99d5b63c7f47aed93e41596ec WatchSource:0}: Error finding container 139b1cf65028018c0794af9130c90de3a34cecb99d5b63c7f47aed93e41596ec: Status 404 returned error can't find the container with id 139b1cf65028018c0794af9130c90de3a34cecb99d5b63c7f47aed93e41596ec
Jul 01 20:46:50.710028 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.710014    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:46:50.826465 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:50.826438    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c4813851e778afe9f5a0f08ee189fd.slice/crio-95861402348b38d3a698044c4c83317b8d73f7bc4d98bc70b8ce93b54cba8988 WatchSource:0}: Error finding container 95861402348b38d3a698044c4c83317b8d73f7bc4d98bc70b8ce93b54cba8988: Status 404 returned error can't find the container with id 95861402348b38d3a698044c4c83317b8d73f7bc4d98bc70b8ce93b54cba8988
Jul 01 20:46:50.878339 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:50.878312    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:51.451853 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.451812    2574 apiserver.go:52] "Watching apiserver"
Jul 01 20:46:51.459741 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.459471    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 20:46:51.460327 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.459909    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7","openshift-multus/multus-additional-cni-plugins-dzdqd","openshift-multus/network-metrics-daemon-9jvqx","openshift-network-diagnostics/network-check-target-8csxj","openshift-ovn-kubernetes/ovnkube-node-7jjmd","kube-system/global-pull-secret-syncer-77s9m","kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal","openshift-cluster-node-tuning-operator/tuned-t5svc","openshift-dns/node-resolver-5h5gr","openshift-image-registry/node-ca-kgp25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal","openshift-multus/multus-fs6qt","openshift-network-operator/iptables-alerter-dm65v","kube-system/konnectivity-agent-2585q"]
Jul 01 20:46:51.462103 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.462077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.466605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.465356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.466605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.466317    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:51.466605 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.466386    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:46:51.473084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.467223    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6khd\""
Jul 01 20:46:51.473084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.467501    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 20:46:51.473084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.467690    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.473084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.467902    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.473084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.471332    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.473648    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473776    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473796    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.473990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.474007    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.474559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.474585    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.474661    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tzl67\""
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.475082    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.476043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.475744    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.477323 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.477100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.478389 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.478365    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.478778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.478759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.479011 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.478991    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zmr4j\""
Jul 01 20:46:51.479251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.479234    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nlb82\""
Jul 01 20:46:51.479438 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.479419    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 20:46:51.479649 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.479632    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.479859 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.479834    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.480533 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.480513    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 20:46:51.481262 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.481242    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.481676 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.481655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.481750 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.481726    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:46:51.482233 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.482019    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.482233 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.482091    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.482363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.482346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 20:46:51.483163 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.483140    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hw5zp\""
Jul 01 20:46:51.485753 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.485731    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 20:46:51.486413 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.486397    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g7j4n\""
Jul 01 20:46:51.486739 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.486688    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 20:46:51.487908 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.487884    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.488031 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488009    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ptqn7\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488304    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.487883    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488577    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488674    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 20:46:51.489216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.488833    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtlw5\""
Jul 01 20:46:51.489696 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.489680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 20:46:51.491187 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.490255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9dd8b\""
Jul 01 20:46:51.491298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-systemd\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.491298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-sys\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.491421 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2x9w\" (UniqueName: \"kubernetes.io/projected/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-kube-api-access-j2x9w\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.491421 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfsh\" (UniqueName: \"kubernetes.io/projected/fc1e123a-a344-4a76-88d2-3c0db03dc084-kube-api-access-xzfsh\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.491421 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-system-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.491571 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-socket-dir-parent\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.491571 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-bin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.491571 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.491571 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e868c3e6-e4a7-4289-9f1a-49d57e56a504-hosts-file\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.491571 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-log-socket\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-netd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-host\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-multus\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-socket-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-systemd-units\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.491849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-sys-fs\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e42e0e9f-1686-4da6-937e-af4809a93f27-host-slash\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-var-lib-kubelet\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.491988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-hostroot\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-etc-kubernetes\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-config\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-dbus\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-os-release\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.492243 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-registration-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-modprobe-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a2f09f4-40c7-410f-a2f4-6fde0343f956-host\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddxx\" (UniqueName: \"kubernetes.io/projected/0a2f09f4-40c7-410f-a2f4-6fde0343f956-kube-api-access-9ddxx\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-k8s-cni-cncf-io\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-netns\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e868c3e6-e4a7-4289-9f1a-49d57e56a504-tmp-dir\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-node-log\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-kubernetes\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfjw\" (UniqueName: \"kubernetes.io/projected/e868c3e6-e4a7-4289-9f1a-49d57e56a504-kube-api-access-lwfjw\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-multus-certs\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-system-cni-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-netns\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.492878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-systemd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysconfig\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-os-release\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-daemon-config\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492943    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-cnibin\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.492976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn98\" (UniqueName: \"kubernetes.io/projected/e42e0e9f-1686-4da6-937e-af4809a93f27-kube-api-access-lhn98\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-var-lib-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-ovn\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-lib-modules\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cnibin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkdq\" (UniqueName: \"kubernetes.io/projected/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-kube-api-access-5qkdq\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-device-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/637e0d63-77d3-46e1-9961-b040febf10f8-kube-api-access-6xf52\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-kubelet\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-tuned\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.493706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk225\" (UniqueName: \"kubernetes.io/projected/be17cc30-0448-48a7-aea8-cc056b04e27e-kube-api-access-rk225\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-env-overrides\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-run\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-tmp\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a2f09f4-40c7-410f-a2f4-6fde0343f956-serviceca\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-bin\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-kubelet\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-conf-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-conf\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cni-binary-copy\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e42e0e9f-1686-4da6-937e-af4809a93f27-iptables-alerter-script\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493771    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-slash\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-etc-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovn-node-metrics-cert\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.494509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-script-lib\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.495819 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.493926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-kubelet-config\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.495819 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.495301    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 20:46:51.499730 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.499116    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:51.520338 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.520307    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcfqt"
Jul 01 20:46:51.529451 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.529431    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcfqt"
Jul 01 20:46:51.548841 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.548800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"a9c4813851e778afe9f5a0f08ee189fd","Type":"ContainerStarted","Data":"95861402348b38d3a698044c4c83317b8d73f7bc4d98bc70b8ce93b54cba8988"}
Jul 01 20:46:51.551460 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.551428    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" event={"ID":"a5e0ed2ae21cf9ca9826a478a311bf2a","Type":"ContainerStarted","Data":"139b1cf65028018c0794af9130c90de3a34cecb99d5b63c7f47aed93e41596ec"}
Jul 01 20:46:51.575629 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.575602    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 20:46:51.595027 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-tmp\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595134 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a2f09f4-40c7-410f-a2f4-6fde0343f956-serviceca\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.595134 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.595134 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.595134 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-bin\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-kubelet\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-conf-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fffcba5b-286c-4bca-9cc5-173f6279f09f-agent-certs\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595246    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-bin\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-conf\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595269    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-kubelet\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cni-binary-copy\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595254    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e42e0e9f-1686-4da6-937e-af4809a93f27-iptables-alerter-script\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.595361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-slash\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-etc-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595404    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-conf\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595408    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovn-node-metrics-cert\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595452    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-etc-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-script-lib\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-kubelet-config\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-slash\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-conf-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-systemd\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595564    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-kubelet-config\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595567    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a2f09f4-40c7-410f-a2f4-6fde0343f956-serviceca\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-sys\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2x9w\" (UniqueName: \"kubernetes.io/projected/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-kube-api-access-j2x9w\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595820    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-systemd\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.595867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595862    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-sys\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595901    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfsh\" (UniqueName: \"kubernetes.io/projected/fc1e123a-a344-4a76-88d2-3c0db03dc084-kube-api-access-xzfsh\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cni-binary-copy\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-system-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-socket-dir-parent\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.595998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-bin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e868c3e6-e4a7-4289-9f1a-49d57e56a504-hosts-file\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-log-socket\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-socket-dir-parent\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-netd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596192    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e42e0e9f-1686-4da6-937e-af4809a93f27-iptables-alerter-script\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-host\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-script-lib\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596262    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-system-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-multus\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.596526 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-socket-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-log-socket\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596350    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e868c3e6-e4a7-4289-9f1a-49d57e56a504-hosts-file\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-systemd-units\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.596370    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596375    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-bin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.596467    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:52.096432224 +0000 UTC m=+3.011710265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-sys-fs\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596539    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e42e0e9f-1686-4da6-937e-af4809a93f27-host-slash\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-var-lib-kubelet\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596493    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-var-lib-cni-multus\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596599    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-host\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-hostroot\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-etc-kubernetes\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596670    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-sys-fs\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-config\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596727    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-cni-netd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596754    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-hostroot\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596770    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-var-lib-kubelet\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-dbus\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e42e0e9f-1686-4da6-937e-af4809a93f27-host-slash\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-os-release\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596868    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-systemd-units\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-registration-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.596972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-modprobe-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597002    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-socket-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597011    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.597778 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597079    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a2f09f4-40c7-410f-a2f4-6fde0343f956-host\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddxx\" (UniqueName: \"kubernetes.io/projected/0a2f09f4-40c7-410f-a2f4-6fde0343f956-kube-api-access-9ddxx\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597135    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-registration-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68e6e793-e805-4c65-8b94-5d540f3bf401-dbus\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-k8s-cni-cncf-io\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597256    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-etc-kubernetes\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-netns\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597222    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-k8s-cni-cncf-io\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-netns\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e868c3e6-e4a7-4289-9f1a-49d57e56a504-tmp-dir\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597500    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-modprobe-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-os-release\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.598661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597532    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-cni-dir\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhsc\" (UniqueName: \"kubernetes.io/projected/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-kube-api-access-rnhsc\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597582    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-node-log\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-kubernetes\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfjw\" (UniqueName: \"kubernetes.io/projected/e868c3e6-e4a7-4289-9f1a-49d57e56a504-kube-api-access-lwfjw\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a2f09f4-40c7-410f-a2f4-6fde0343f956-host\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597624    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovnkube-config\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597683    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-node-log\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-multus-certs\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597731    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e868c3e6-e4a7-4289-9f1a-49d57e56a504-tmp-dir\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-system-cni-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-kubernetes\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597786    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be17cc30-0448-48a7-aea8-cc056b04e27e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-netns\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-host-run-multus-certs\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-systemd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.599523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysconfig\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597885    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-systemd\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597912    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-run-netns\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-system-cni-dir\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-os-release\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597977    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysconfig\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.597980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-daemon-config\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-cnibin\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598059    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-os-release\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-sysctl-d\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn98\" (UniqueName: \"kubernetes.io/projected/e42e0e9f-1686-4da6-937e-af4809a93f27-kube-api-access-lhn98\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-var-lib-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-ovn\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598183    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be17cc30-0448-48a7-aea8-cc056b04e27e-cnibin\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-lib-modules\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-run-ovn\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.600580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cnibin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598280    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-lib-modules\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598309    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-var-lib-openvswitch\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-cnibin\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598501    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-multus-daemon-config\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkdq\" (UniqueName: \"kubernetes.io/projected/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-kube-api-access-5qkdq\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598541    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-device-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/637e0d63-77d3-46e1-9961-b040febf10f8-kube-api-access-6xf52\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598626    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637e0d63-77d3-46e1-9961-b040febf10f8-device-dir\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-kubelet\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-tuned\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598731    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-host-kubelet\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk225\" (UniqueName: \"kubernetes.io/projected/be17cc30-0448-48a7-aea8-cc056b04e27e-kube-api-access-rk225\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.598819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fffcba5b-286c-4bca-9cc5-173f6279f09f-konnectivity-ca\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.599396    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-env-overrides\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.599634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-env-overrides\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.599678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-run\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.601611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.599757    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc1e123a-a344-4a76-88d2-3c0db03dc084-run\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.602403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.601832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-tmp\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.602403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.601902    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-ovn-node-metrics-cert\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.602403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.602336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc1e123a-a344-4a76-88d2-3c0db03dc084-etc-tuned\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.603283 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.603125    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:51.603283 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.603151    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:51.603283 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.603163    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:51.603283 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.603237    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:52.103218707 +0000 UTC m=+3.018496725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:51.606863 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.606629    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2x9w\" (UniqueName: \"kubernetes.io/projected/a6412b14-0c6f-48e8-b6c8-6011fe7228a7-kube-api-access-j2x9w\") pod \"ovnkube-node-7jjmd\" (UID: \"a6412b14-0c6f-48e8-b6c8-6011fe7228a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.606863 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.606699    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfsh\" (UniqueName: \"kubernetes.io/projected/fc1e123a-a344-4a76-88d2-3c0db03dc084-kube-api-access-xzfsh\") pod \"tuned-t5svc\" (UID: \"fc1e123a-a344-4a76-88d2-3c0db03dc084\") " pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.607643 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.607398    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:51.610860 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.610803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf52\" (UniqueName: \"kubernetes.io/projected/637e0d63-77d3-46e1-9961-b040febf10f8-kube-api-access-6xf52\") pod \"aws-ebs-csi-driver-node-nw9b7\" (UID: \"637e0d63-77d3-46e1-9961-b040febf10f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.611313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.611289    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn98\" (UniqueName: \"kubernetes.io/projected/e42e0e9f-1686-4da6-937e-af4809a93f27-kube-api-access-lhn98\") pod \"iptables-alerter-dm65v\" (UID: \"e42e0e9f-1686-4da6-937e-af4809a93f27\") " pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.611936 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.611890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfjw\" (UniqueName: \"kubernetes.io/projected/e868c3e6-e4a7-4289-9f1a-49d57e56a504-kube-api-access-lwfjw\") pod \"node-resolver-5h5gr\" (UID: \"e868c3e6-e4a7-4289-9f1a-49d57e56a504\") " pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.612142 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.612118    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk225\" (UniqueName: \"kubernetes.io/projected/be17cc30-0448-48a7-aea8-cc056b04e27e-kube-api-access-rk225\") pod \"multus-additional-cni-plugins-dzdqd\" (UID: \"be17cc30-0448-48a7-aea8-cc056b04e27e\") " pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.612225 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.612204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddxx\" (UniqueName: \"kubernetes.io/projected/0a2f09f4-40c7-410f-a2f4-6fde0343f956-kube-api-access-9ddxx\") pod \"node-ca-kgp25\" (UID: \"0a2f09f4-40c7-410f-a2f4-6fde0343f956\") " pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.612855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.612774    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkdq\" (UniqueName: \"kubernetes.io/projected/6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd-kube-api-access-5qkdq\") pod \"multus-fs6qt\" (UID: \"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd\") " pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.699975 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.699939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.700162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.699998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fffcba5b-286c-4bca-9cc5-173f6279f09f-agent-certs\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.700162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.700059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhsc\" (UniqueName: \"kubernetes.io/projected/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-kube-api-access-rnhsc\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.700162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.700104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fffcba5b-286c-4bca-9cc5-173f6279f09f-konnectivity-ca\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.700162 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.700154    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:51.700380 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:51.700220    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:46:52.200199757 +0000 UTC m=+3.115477775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:51.700727 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.700704    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fffcba5b-286c-4bca-9cc5-173f6279f09f-konnectivity-ca\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.703418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.703349    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fffcba5b-286c-4bca-9cc5-173f6279f09f-agent-certs\") pod \"konnectivity-agent-2585q\" (UID: \"fffcba5b-286c-4bca-9cc5-173f6279f09f\") " pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:51.713388 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.713366    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhsc\" (UniqueName: \"kubernetes.io/projected/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-kube-api-access-rnhsc\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:51.795940 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.795900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm65v"
Jul 01 20:46:51.802710 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.802686    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzdqd"
Jul 01 20:46:51.813782 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.813753    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t5svc"
Jul 01 20:46:51.821002 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.820975    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7"
Jul 01 20:46:51.827894 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.827872    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:46:51.835851 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.835830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5h5gr"
Jul 01 20:46:51.845896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.845863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kgp25"
Jul 01 20:46:51.853933 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.853911    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fs6qt"
Jul 01 20:46:51.860920 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:51.860900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:46:52.102440 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:52.102358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:52.102586 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.102479    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:52.102586 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.102536    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:53.102521535 +0000 UTC m=+4.017799552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:52.202729 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:52.202672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:52.202910 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:52.202751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:52.202910 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.202861    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:52.203040 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.202917    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:52.203040 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.202941    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:52.203040 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.202950    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:46:53.202926897 +0000 UTC m=+4.118204935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:52.203040 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.202954    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:52.203040 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:52.203010    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:53.202992669 +0000 UTC m=+4.118270691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:52.531807 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:52.531754    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:51 +0000 UTC" deadline="2027-12-25 18:17:27.50204022 +0000 UTC"
Jul 01 20:46:52.531807 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:52.531791    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13005h30m34.970253593s"
Jul 01 20:46:52.821231 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:52.821207    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e0e9f_1686_4da6_937e_af4809a93f27.slice/crio-dc6a37e00a6af8e0792cd313aea1821dd9e1b09c3b677e97a50032d821cdec19 WatchSource:0}: Error finding container dc6a37e00a6af8e0792cd313aea1821dd9e1b09c3b677e97a50032d821cdec19: Status 404 returned error can't find the container with id dc6a37e00a6af8e0792cd313aea1821dd9e1b09c3b677e97a50032d821cdec19
Jul 01 20:46:52.823156 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:52.823124    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1e123a_a344_4a76_88d2_3c0db03dc084.slice/crio-1dada4aaa6f5f8c9671083cd631a417ec9bcdc93ebd4958fae0a2c73f22db334 WatchSource:0}: Error finding container 1dada4aaa6f5f8c9671083cd631a417ec9bcdc93ebd4958fae0a2c73f22db334: Status 404 returned error can't find the container with id 1dada4aaa6f5f8c9671083cd631a417ec9bcdc93ebd4958fae0a2c73f22db334
Jul 01 20:46:52.826041 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:52.826017    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6412b14_0c6f_48e8_b6c8_6011fe7228a7.slice/crio-5dffbcaf358c074ea6d016693aee16b890c88c213b34c6b3d814153a55b9b5fc WatchSource:0}: Error finding container 5dffbcaf358c074ea6d016693aee16b890c88c213b34c6b3d814153a55b9b5fc: Status 404 returned error can't find the container with id 5dffbcaf358c074ea6d016693aee16b890c88c213b34c6b3d814153a55b9b5fc
Jul 01 20:46:52.831684 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:52.831613    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fac2d09_f4b9_46b4_85a1_cf5f71bd34dd.slice/crio-c143f9013a8b768e4ef84d2961f07e83766e6bd9366ae5452f54954dc009f1f8 WatchSource:0}: Error finding container c143f9013a8b768e4ef84d2961f07e83766e6bd9366ae5452f54954dc009f1f8: Status 404 returned error can't find the container with id c143f9013a8b768e4ef84d2961f07e83766e6bd9366ae5452f54954dc009f1f8
Jul 01 20:46:52.832504 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:46:52.832479    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2f09f4_40c7_410f_a2f4_6fde0343f956.slice/crio-5167a7e4d4bab96f824ecf2841a2a12daf4a0d83436968f75d585732e383d3a8 WatchSource:0}: Error finding container 5167a7e4d4bab96f824ecf2841a2a12daf4a0d83436968f75d585732e383d3a8: Status 404 returned error can't find the container with id 5167a7e4d4bab96f824ecf2841a2a12daf4a0d83436968f75d585732e383d3a8
Jul 01 20:46:53.113739 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.113576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:53.113959 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.113775    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:53.113959 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.113854    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:55.113831548 +0000 UTC m=+6.029109569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:53.214735 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.214686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:53.214880 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.214751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:53.214880 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214838    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:53.214951 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214879    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:53.214951 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214895    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:53.214951 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214909    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:46:55.214890153 +0000 UTC m=+6.130168178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:53.214951 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214909    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:53.215084 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.214956    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:55.214943688 +0000 UTC m=+6.130221720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:53.533142 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.532970    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:51 +0000 UTC" deadline="2028-03-30 02:43:53.187598372 +0000 UTC"
Jul 01 20:46:53.533142 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.533035    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15293h56m59.654568134s"
Jul 01 20:46:53.545855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.545826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:53.546051 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.545976    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:46:53.546423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.546406    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:53.546730 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.546501    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:46:53.546730 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.546564    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:53.546730 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:53.546635    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:46:53.573027 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.572993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"5dffbcaf358c074ea6d016693aee16b890c88c213b34c6b3d814153a55b9b5fc"}
Jul 01 20:46:53.584713 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.584674    2574 generic.go:356] "Generic (PLEG): container finished" podID="a9c4813851e778afe9f5a0f08ee189fd" containerID="cd3c61249bece10eb4baddd71aa24b0ead35ba4d3031faa3db48f965579330ec" exitCode=0
Jul 01 20:46:53.584851 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.584781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"a9c4813851e778afe9f5a0f08ee189fd","Type":"ContainerDied","Data":"cd3c61249bece10eb4baddd71aa24b0ead35ba4d3031faa3db48f965579330ec"}
Jul 01 20:46:53.603075 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.603002    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" event={"ID":"a5e0ed2ae21cf9ca9826a478a311bf2a","Type":"ContainerStarted","Data":"a34a4e1111a342c381a1e7f717e97110e2a77825519e4921b7ff6870e6c9f634"}
Jul 01 20:46:53.627214 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.627113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerStarted","Data":"462981683d731ed975e03b1dedb0dd455a8273422cb38ae601bb869d9ae8a055"}
Jul 01 20:46:53.639899 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.639757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7" event={"ID":"637e0d63-77d3-46e1-9961-b040febf10f8","Type":"ContainerStarted","Data":"4d6a8b6e21d327b9c693bd23982f9f791c0b41697aed71a78c65c103fb74267d"}
Jul 01 20:46:53.654428 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.654394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2585q" event={"ID":"fffcba5b-286c-4bca-9cc5-173f6279f09f","Type":"ContainerStarted","Data":"99a5f9e01212bc177e705f42d9e69d1875ab1b2ed555259afbe15803efac7e54"}
Jul 01 20:46:53.663217 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.660711    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fs6qt" event={"ID":"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd","Type":"ContainerStarted","Data":"c143f9013a8b768e4ef84d2961f07e83766e6bd9366ae5452f54954dc009f1f8"}
Jul 01 20:46:53.666923 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.666893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t5svc" event={"ID":"fc1e123a-a344-4a76-88d2-3c0db03dc084","Type":"ContainerStarted","Data":"1dada4aaa6f5f8c9671083cd631a417ec9bcdc93ebd4958fae0a2c73f22db334"}
Jul 01 20:46:53.671949 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.671920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm65v" event={"ID":"e42e0e9f-1686-4da6-937e-af4809a93f27","Type":"ContainerStarted","Data":"dc6a37e00a6af8e0792cd313aea1821dd9e1b09c3b677e97a50032d821cdec19"}
Jul 01 20:46:53.693874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.693808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5h5gr" event={"ID":"e868c3e6-e4a7-4289-9f1a-49d57e56a504","Type":"ContainerStarted","Data":"5fc2849c213ce161159173a6c62a3d0e288ecf03067d429bc7055110a37c5f46"}
Jul 01 20:46:53.700728 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:53.700695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kgp25" event={"ID":"0a2f09f4-40c7-410f-a2f4-6fde0343f956","Type":"ContainerStarted","Data":"5167a7e4d4bab96f824ecf2841a2a12daf4a0d83436968f75d585732e383d3a8"}
Jul 01 20:46:54.712842 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:54.712808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"a9c4813851e778afe9f5a0f08ee189fd","Type":"ContainerStarted","Data":"e135825fd370a2543941f30c8d71ecd6ab632fcfde960e64b8c66d2edb0331e5"}
Jul 01 20:46:54.729015 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:54.728943    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" podStartSLOduration=4.728923057 podStartE2EDuration="4.728923057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:53.625003345 +0000 UTC m=+4.540281391" watchObservedRunningTime="2026-07-01 20:46:54.728923057 +0000 UTC m=+5.644201098"
Jul 01 20:46:55.134631 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.134546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:55.134881 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.134688    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:55.134881 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.134770    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.134751565 +0000 UTC m=+10.050029582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:55.235792 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.235758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:55.235971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.235800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:55.235971 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.235928    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:55.235971 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.235948    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:55.235971 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.235960    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:55.236159 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.236014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.235994274 +0000 UTC m=+10.151272291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:55.236385 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.236366    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:55.236474 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.236416    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.23640268 +0000 UTC m=+10.151680698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.550505    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.550633    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.550989    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.551080    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:55.551123    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:55.551605 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:55.551187    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:57.543793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:57.543913    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:57.543955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:57.544060    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:57.543793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:57.544274 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:57.544162    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:46:59.168759 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.168691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:59.169209 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.168843    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:59.169209 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.168923    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:07.168902767 +0000 UTC m=+18.084180796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:59.270077 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.270040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:59.270247 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.270097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:59.270314 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270271    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:59.270314 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270290    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:59.270314 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270303    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:59.270418 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270360    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:07.270341213 +0000 UTC m=+18.185619230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:59.270781 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270762    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:59.270899 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.270814    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:47:07.270799255 +0000 UTC m=+18.186077274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:59.550260 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.550182    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:46:59.550417 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.550297    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:46:59.550850 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.550668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:46:59.550850 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.550776    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:46:59.551268 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:46:59.551127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:46:59.551268 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:46:59.551223    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:01.543971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:01.543875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:01.543971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:01.543906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:01.543971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:01.543875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:01.544489 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:01.543997    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:01.544489 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:01.544045    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:01.544489 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:01.544117    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:03.543239 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:03.543208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:03.543239 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:03.543222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:03.543239 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:03.543229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:03.543759 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:03.543318    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:03.543759 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:03.543372    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:03.543759 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:03.543443    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:05.543171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:05.543129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:05.543171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:05.543153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:05.543639 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:05.543129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:05.543639 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:05.543247    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:05.543639 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:05.543354    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:05.543639 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:05.543455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:07.228301 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.228258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:07.228794 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.228413    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:07.228794 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.228465    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:23.228452038 +0000 UTC m=+34.143730054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:07.328889 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.328856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:07.329061 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.328898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:07.329061 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329029    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:47:07.329061 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329029    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:07.329061 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329048    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:47:07.329061 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329066    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:07.329279 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329116    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:47:23.329095099 +0000 UTC m=+34.244373117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:07.329279 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.329135    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:23.329124907 +0000 UTC m=+34.244402930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:07.543258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.543192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:07.543398 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.543192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:07.543398 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.543302    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:07.543398 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:07.543205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:07.543546 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.543359    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:07.543546 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:07.543476    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:09.545008 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:09.544972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:09.545456 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:09.545079    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:09.545456 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:09.545123    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:09.545456 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:09.545210    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:09.545456 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:09.545252    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:09.545456 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:09.545300    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:11.543001 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:11.542960    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:11.543529 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:11.542960    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:11.543529 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:11.543095    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:11.543529 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:11.543194    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:11.543529 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:11.542969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:11.543529 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:11.543304    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:12.751615 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.751223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kgp25" event={"ID":"0a2f09f4-40c7-410f-a2f4-6fde0343f956","Type":"ContainerStarted","Data":"1f399be983688ea42b484834a6e60a500c2ace71b396a9fc785b49c54ea9e768"}
Jul 01 20:47:12.754250 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754232    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:47:12.754569 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754545    2574 generic.go:356] "Generic (PLEG): container finished" podID="a6412b14-0c6f-48e8-b6c8-6011fe7228a7" containerID="0aa44f124f84257542c99eef541aa7b29cf328f937b0b89d1b0385a4612f007a" exitCode=1
Jul 01 20:47:12.754664 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"d1e7f6228086978225e2f15a7c24d37f145b2df80a7add5c4d09185ab93c984a"}
Jul 01 20:47:12.754736 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754702    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"832bc31792b2ac13a40d626505bd3ceec1d4789642fff172ef1ccfe5ca86d5db"}
Jul 01 20:47:12.754787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"ef97188628f0d574d0999ba486c31c33c1f05087768722df08b9be94c5396a82"}
Jul 01 20:47:12.754840 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754826    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"8228ff04050143cb2b4dd8e600cabc53d71be19f93665d7b060065ec0dc5b76e"}
Jul 01 20:47:12.754888 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754878    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerDied","Data":"0aa44f124f84257542c99eef541aa7b29cf328f937b0b89d1b0385a4612f007a"}
Jul 01 20:47:12.754943 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.754927    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"607963ec38521a51d48e6d42b0bc9b7d5ab369e34f3d6b137e1bcb00cebf601f"}
Jul 01 20:47:12.756229 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.756208    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="5bf72fca30c688e502aadd0ba6a6a3c8e5eeccb94e1de71c7fc5d2e798dee57f" exitCode=0
Jul 01 20:47:12.756437 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.756280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"5bf72fca30c688e502aadd0ba6a6a3c8e5eeccb94e1de71c7fc5d2e798dee57f"}
Jul 01 20:47:12.757871 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.757844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7" event={"ID":"637e0d63-77d3-46e1-9961-b040febf10f8","Type":"ContainerStarted","Data":"88a69aa0cfde39c1eb1f04a229dffff82b2754a12fde1ddc5b202e5a111d4028"}
Jul 01 20:47:12.759241 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.759218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2585q" event={"ID":"fffcba5b-286c-4bca-9cc5-173f6279f09f","Type":"ContainerStarted","Data":"d0f6bf6cc8a635c3d8104baf9adfe52c817d335195ae3f746915f964f2736e15"}
Jul 01 20:47:12.760695 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.760673    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fs6qt" event={"ID":"6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd","Type":"ContainerStarted","Data":"8a78b5bb7de542d605a2bd2b331c226a7b24dc6096b317a39e2c8f26c52726db"}
Jul 01 20:47:12.762826 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.762800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t5svc" event={"ID":"fc1e123a-a344-4a76-88d2-3c0db03dc084","Type":"ContainerStarted","Data":"56877922bcb82925dd9f00c334ecee44b39159c2af9b68004f27afe348f8b7ae"}
Jul 01 20:47:12.766340 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.766317    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5h5gr" event={"ID":"e868c3e6-e4a7-4289-9f1a-49d57e56a504","Type":"ContainerStarted","Data":"bb582e908e69d29ba3425f21d3fbd5c6e0ef3162ab835df2a3f8391422b5417e"}
Jul 01 20:47:12.768154 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.768114    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kgp25" podStartSLOduration=3.8934722920000002 podStartE2EDuration="22.76810301s" totalImagesPullingTime="18.874630718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:12.767755836 +0000 UTC m=+23.683033878" watchObservedRunningTime="2026-07-01 20:47:12.76810301 +0000 UTC m=+23.683381048"
Jul 01 20:47:12.768257 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.768192    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" podStartSLOduration=22.768186779 podStartE2EDuration="22.768186779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:54.729334407 +0000 UTC m=+5.644612446" watchObservedRunningTime="2026-07-01 20:47:12.768186779 +0000 UTC m=+23.683464794"
Jul 01 20:47:12.786282 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.786244    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fs6qt" podStartSLOduration=3.852555853 podStartE2EDuration="22.786232708s" totalImagesPullingTime="18.933676855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:12.786006014 +0000 UTC m=+23.701284076" watchObservedRunningTime="2026-07-01 20:47:12.786232708 +0000 UTC m=+23.701510745"
Jul 01 20:47:12.849948 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.849897    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t5svc" podStartSLOduration=3.963396912 podStartE2EDuration="22.849880968s" totalImagesPullingTime="18.886484056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:12.830898131 +0000 UTC m=+23.746176169" watchObservedRunningTime="2026-07-01 20:47:12.849880968 +0000 UTC m=+23.765159024"
Jul 01 20:47:12.850205 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.850183    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2585q" podStartSLOduration=11.99817078 podStartE2EDuration="22.850178245s" totalImagesPullingTime="10.852007465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:12.849423712 +0000 UTC m=+23.764701750" watchObservedRunningTime="2026-07-01 20:47:12.850178245 +0000 UTC m=+23.765456283"
Jul 01 20:47:12.875886 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.875851    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:12.954297 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:12.954261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:13.054962 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.054867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:13.349665 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.349638    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 20:47:13.427095 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.427064    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:13.538875 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.538839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:13.543076 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.543051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:13.543201 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:13.543149    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:13.543201 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.543051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:13.543280 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:13.543218    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:13.543280 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.543237    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:13.543280 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:13.543276    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:13.547558 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.547492    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T20:47:13.349658613Z","UUID":"5af4a55d-29ac-47d5-b736-3cee08a060e3","Handler":null,"Name":"","Endpoint":""}
Jul 01 20:47:13.548840 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.548825    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 20:47:13.548916 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.548871    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 20:47:13.770684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.770647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7" event={"ID":"637e0d63-77d3-46e1-9961-b040febf10f8","Type":"ContainerStarted","Data":"422d6a8aac71272c7e8ea20d8f91a9e3b162bfdb39326484eef2df1773ec409a"}
Jul 01 20:47:13.772281 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.772231    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm65v" event={"ID":"e42e0e9f-1686-4da6-937e-af4809a93f27","Type":"ContainerStarted","Data":"21a373abb7eeb2c37f058aa0dc386cecef6ccef61e1a97c5c0cd929e01b31722"}
Jul 01 20:47:13.789140 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.789089    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5h5gr" podStartSLOduration=4.915085705 podStartE2EDuration="23.789068329s" totalImagesPullingTime="18.873982624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:12.869543515 +0000 UTC m=+23.784821554" watchObservedRunningTime="2026-07-01 20:47:13.789068329 +0000 UTC m=+24.704346364"
Jul 01 20:47:13.789349 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.789320    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dm65v" podStartSLOduration=5.903300757 podStartE2EDuration="24.789311026s" totalImagesPullingTime="18.886010269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:13.789057876 +0000 UTC m=+24.704335916" watchObservedRunningTime="2026-07-01 20:47:13.789311026 +0000 UTC m=+24.704589067"
Jul 01 20:47:13.804171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.804125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:13.904993 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:13.904957    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:14.016582 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.016550    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:14.355133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.355098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:14.778369 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.778341    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:47:14.778815 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.778748    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"8aac8be20b50fff790664c0b3b9b70fe003106dedf058f1033ef5a45f88af80e"}
Jul 01 20:47:14.781079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.781049    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7" event={"ID":"637e0d63-77d3-46e1-9961-b040febf10f8","Type":"ContainerStarted","Data":"e08cd1bc3927f676351f934391b3bcae5ad8744a594c88785b97570d4f2270f5"}
Jul 01 20:47:14.806419 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:14.806360    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nw9b7" podStartSLOduration=3.508289587 podStartE2EDuration="24.806340937s" totalImagesPullingTime="21.29805135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:14.80591638 +0000 UTC m=+25.721194419" watchObservedRunningTime="2026-07-01 20:47:14.806340937 +0000 UTC m=+25.721618977"
Jul 01 20:47:15.514026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.513997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:15.543273 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.543220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:15.543428 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.543220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:15.543428 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:15.543355    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:15.543428 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.543220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:15.543428 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:15.543395    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:15.543569 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:15.543491    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:15.618055 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.618019    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:15.715072 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.714914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:15.851921 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.851848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:15.983244 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:15.983207    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:16.205276 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.205236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:16.605847 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.605768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:16.662465 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.662429    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:47:16.663193 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.663173    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:47:16.677171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.677129    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:16.785849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.785822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:47:16.786365 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:16.786340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2585q"
Jul 01 20:47:17.207122 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.207079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:17.402974 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.402947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:17.543568 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.543409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:17.543781 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.543417    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:17.543781 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:17.543668    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:17.543904 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:17.543782    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:17.543904 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.543418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:17.543904 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:17.543881    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:17.786527 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.786496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:17.789082 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.789052    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerStarted","Data":"e876ca35033f0a503c8c737cbf09443848ed172dda2b23f5ab26b56443a68e4f"}
Jul 01 20:47:17.792333 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.792290    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:47:17.792679 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.792657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"6b742cfef16e25640304602a098d1c7cd6eaa02e1ac561a64fcc646469884a30"}
Jul 01 20:47:17.792985 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.792953    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:17.793077 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.793028    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:17.793174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.793155    2574 scope.go:117] "RemoveContainer" containerID="0aa44f124f84257542c99eef541aa7b29cf328f937b0b89d1b0385a4612f007a"
Jul 01 20:47:17.809408 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.809389    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:17.813292 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.813273    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:17.853533 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:17.853505    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.017188 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.017096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.194118 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.194085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.409380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.409298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.509558 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.509532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.610388 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.610346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:18.796162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.796129    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="e876ca35033f0a503c8c737cbf09443848ed172dda2b23f5ab26b56443a68e4f" exitCode=0
Jul 01 20:47:18.796392 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.796214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"e876ca35033f0a503c8c737cbf09443848ed172dda2b23f5ab26b56443a68e4f"}
Jul 01 20:47:18.800121 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.800107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:47:18.800469 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.800449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" event={"ID":"a6412b14-0c6f-48e8-b6c8-6011fe7228a7","Type":"ContainerStarted","Data":"3d7cacccf290f5b5486f7ef58db6aa5afeca44dbe5710ab3c93f8757956a6b95"}
Jul 01 20:47:18.800541 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.800523    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 01 20:47:18.865410 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:18.863319    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd" podStartSLOduration=9.915112603 podStartE2EDuration="28.863298976s" totalImagesPullingTime="18.948186373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:18.859481596 +0000 UTC m=+29.774759634" watchObservedRunningTime="2026-07-01 20:47:18.863298976 +0000 UTC m=+29.778577014"
Jul 01 20:47:19.013820 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.013790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:19.202497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.202464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:19.319792 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.319743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:19.441300 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.441263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:19.544285 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.544215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:19.544418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.544294    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:19.544418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.544337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:19.544418 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:19.544365    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:19.544418 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:19.544409    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:19.544635 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:19.544484    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:19.547544 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.547526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:19.599574 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:19.599546    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:20.227765 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:20.227727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:20.314900 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:20.314866    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:20.617185 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:20.617113    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:20.806214 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:20.806184    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="4337d653653b03ab7dfc41fe84b031e97fe681dd2e7bc9344c7a5fc6e8110112" exitCode=0
Jul 01 20:47:20.806373 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:20.806274    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"4337d653653b03ab7dfc41fe84b031e97fe681dd2e7bc9344c7a5fc6e8110112"}
Jul 01 20:47:21.047976 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.047945    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:21.220136 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.220106    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:21.544111 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.544083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:21.544298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.544083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:21.544298 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:21.544223    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:21.544298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.544233    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:21.544462 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:21.544313    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:21.544462 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:21.544381    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:21.611464 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.611434    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:21.815430 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:21.815361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:22.433107 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:22.433036    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:22.609350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:22.609317    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:22.812356 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:22.812325    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="e3f1c36c422a902d993bf3e0f18a63402e847ff59149889273d800f8a29f217c" exitCode=0
Jul 01 20:47:22.812509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:22.812368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"e3f1c36c422a902d993bf3e0f18a63402e847ff59149889273d800f8a29f217c"}
Jul 01 20:47:23.006499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.006470    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:23.233112 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.233082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:23.250156 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.250132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:23.250298 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.250249    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:23.250356 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.250300    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret podName:68e6e793-e805-4c65-8b94-5d540f3bf401 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:55.250287023 +0000 UTC m=+66.165565038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret") pod "global-pull-secret-syncer-77s9m" (UID: "68e6e793-e805-4c65-8b94-5d540f3bf401") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:23.350632 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.350606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:23.350788 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.350647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:23.350788 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350781    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:47:23.350897 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350784    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:23.350897 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350795    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:47:23.350897 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350810    2574 projected.go:196] Error preparing data for projected volume kube-api-access-b9bc5 for pod openshift-network-diagnostics/network-check-target-8csxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:23.350897 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350850    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs podName:4c08e116-b49a-4f87-a27c-2e4a5b0cfece nodeName:}" failed. No retries permitted until 2026-07-01 20:47:55.350830897 +0000 UTC m=+66.266108913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs") pod "network-metrics-daemon-9jvqx" (UID: "4c08e116-b49a-4f87-a27c-2e4a5b0cfece") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:23.350897 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.350870    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5 podName:2eb44aee-dfbc-45b7-9ee4-45ee9acbd117 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:55.350860169 +0000 UTC m=+66.266138191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9bc5" (UniqueName: "kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5") pod "network-check-target-8csxj" (UID: "2eb44aee-dfbc-45b7-9ee4-45ee9acbd117") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:23.544012 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.543941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:23.544162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.543943    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:23.544162 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.544071    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:23.544162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.544077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:23.544338 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.544198    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:23.544338 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:23.544276    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:23.606299 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:23.606272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:24.398948 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:24.398906    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:24.606631 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:24.606570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:24.703675 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:24.703643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:24.994528 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:24.994440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:25.208318 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.208288    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.543823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.543834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:25.543948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:25.544029    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.544097    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:25.544334 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:25.544188    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:25.624731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.624674    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:25.997972 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:25.997937    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:26.219461 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:26.219404    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:26.386464 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:26.386375    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:26.823813 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:26.823790    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerStarted","Data":"2887d7bdb2be7a5608431e4151ffd1766920fda24fe7217fbe9bd17f63df06b2"}
Jul 01 20:47:27.543978 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:27.543948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:27.543978 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:27.543966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:27.544203 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:27.543948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:27.544203 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:27.544040    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:27.544203 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:27.544117    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:27.544313 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:27.544199    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:27.831212 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:27.831141    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="2887d7bdb2be7a5608431e4151ffd1766920fda24fe7217fbe9bd17f63df06b2" exitCode=0
Jul 01 20:47:27.831212 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:27.831191    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"2887d7bdb2be7a5608431e4151ffd1766920fda24fe7217fbe9bd17f63df06b2"}
Jul 01 20:47:28.836111 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:28.836073    2574 generic.go:356] "Generic (PLEG): container finished" podID="be17cc30-0448-48a7-aea8-cc056b04e27e" containerID="b5b1cef484b72f62be1ee8357477d4ab694f0c0697f35007f5f8db6a68c5095c" exitCode=0
Jul 01 20:47:28.836589 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:28.836140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerDied","Data":"b5b1cef484b72f62be1ee8357477d4ab694f0c0697f35007f5f8db6a68c5095c"}
Jul 01 20:47:29.544067 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:29.543874    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:29.544241 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:29.543916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:29.544241 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:29.544149    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:29.544241 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:29.543941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:29.544406 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:29.544239    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:29.544406 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:29.544322    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:29.840814 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:29.840745    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" event={"ID":"be17cc30-0448-48a7-aea8-cc056b04e27e","Type":"ContainerStarted","Data":"43948f3af6e6ed63cd4214b77b28c07b6028734246537ab3154e5dea6b86608a"}
Jul 01 20:47:29.866451 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:29.866413    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dzdqd" podStartSLOduration=19.133029199 podStartE2EDuration="40.866401884s" totalImagesPullingTime="21.733372685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:29.866033266 +0000 UTC m=+40.781311303" watchObservedRunningTime="2026-07-01 20:47:29.866401884 +0000 UTC m=+40.781679922"
Jul 01 20:47:31.178570 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.178534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:31.311878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.311843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:31.475392 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.475363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:31.543416 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.543384    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:31.543416 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.543399    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:31.543416 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.543384    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:31.543667 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:31.543489    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:31.543667 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:31.543551    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:31.543667 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:31.543597    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:31.589366 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.589341    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:31.691584 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.691560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:31.832818 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:31.832743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:32.601040 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:32.601008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:32.709499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:32.709472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:32.845384 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:32.845347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:32.993619 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:32.993592    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:33.142871 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.142842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:33.305079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.305012    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:33.543465 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.543439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:33.543465 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.543459    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:33.543670 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.543446    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:33.543670 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:33.543542    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:33.543670 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:33.543611    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:33.543796 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:33.543680    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:33.778551 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.778519    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:33.853960 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:33.853933    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:34.454916 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.454861    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77s9m"]
Jul 01 20:47:34.455053 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.455006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:34.455145 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:34.455121    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:34.459828 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.459805    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jvqx"]
Jul 01 20:47:34.459931 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.459917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:34.460039 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:34.460015    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:34.468001 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.467980    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8csxj"]
Jul 01 20:47:34.468107 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.468072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:34.468181 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:34.468156    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:34.528673 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.528608    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:34.617454 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:34.617428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:36.543208 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:36.543174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:36.543208 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:36.543195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:36.543208 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:36.543204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:36.544048 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:36.543286    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8csxj" podUID="2eb44aee-dfbc-45b7-9ee4-45ee9acbd117"
Jul 01 20:47:36.544048 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:36.543405    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jvqx" podUID="4c08e116-b49a-4f87-a27c-2e4a5b0cfece"
Jul 01 20:47:36.544048 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:36.543465    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77s9m" podUID="68e6e793-e805-4c65-8b94-5d540f3bf401"
Jul 01 20:47:37.978858 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:37.978694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:38.059541 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.059496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:38.426648 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.426627    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeReady"
Jul 01 20:47:38.426776 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.426760    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 20:47:38.481504 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.481482    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6jx6"]
Jul 01 20:47:38.498293 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.498267    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ktjs7"]
Jul 01 20:47:38.498397 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.498305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.501233 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.501206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 20:47:38.501615 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.501600    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 20:47:38.502007 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.501990    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-48t77\""
Jul 01 20:47:38.502264 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.502252    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 20:47:38.510132 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.510113    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6jx6"]
Jul 01 20:47:38.510233 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.510204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.512139 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.512082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ktjs7"]
Jul 01 20:47:38.513390 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.513357    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z5zdv\""
Jul 01 20:47:38.513848 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.513826    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 20:47:38.514174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.514157    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 20:47:38.514174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.514168    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 20:47:38.514676 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.514661    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 20:47:38.525221 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.525205    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fnjkr"]
Jul 01 20:47:38.541423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.541361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.543198 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.543168    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:38.543286 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.543275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:38.543681 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.543450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:38.544636 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.544618    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnjkr"]
Jul 01 20:47:38.546578 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.546559    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 20:47:38.547177 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547158    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:38.547534 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547510    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64glk\""
Jul 01 20:47:38.547622 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547573    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:47:38.547684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547575    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fncc2\""
Jul 01 20:47:38.547907 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 20:47:38.547907 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547822    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h98zr\""
Jul 01 20:47:38.547907 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.547894    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:47:38.548126 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.548111    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:38.566290 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba42c95-d563-4e39-b778-6a76b43761e3-metrics-tls\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.566380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsjf\" (UniqueName: \"kubernetes.io/projected/8ba42c95-d563-4e39-b778-6a76b43761e3-kube-api-access-wfsjf\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.566380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltf2\" (UniqueName: \"kubernetes.io/projected/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-api-access-4ltf2\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.566380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba42c95-d563-4e39-b778-6a76b43761e3-config-volume\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.566380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-crio-socket\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.566380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-data-volume\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.566594 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7jn\" (UniqueName: \"kubernetes.io/projected/464150bb-e3fb-42f5-881e-a3e4f64b08f4-kube-api-access-vd7jn\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.566594 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.566594 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.566594 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464150bb-e3fb-42f5-881e-a3e4f64b08f4-cert\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.566594 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.566564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba42c95-d563-4e39-b778-6a76b43761e3-tmp-dir\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.666992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-crio-socket\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667138 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-data-volume\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667138 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7jn\" (UniqueName: \"kubernetes.io/projected/464150bb-e3fb-42f5-881e-a3e4f64b08f4-kube-api-access-vd7jn\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.667138 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667287 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667287 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667188    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-crio-socket\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667287 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464150bb-e3fb-42f5-881e-a3e4f64b08f4-cert\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.667287 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba42c95-d563-4e39-b778-6a76b43761e3-tmp-dir\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667488 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba42c95-d563-4e39-b778-6a76b43761e3-metrics-tls\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667488 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsjf\" (UniqueName: \"kubernetes.io/projected/8ba42c95-d563-4e39-b778-6a76b43761e3-kube-api-access-wfsjf\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667488 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltf2\" (UniqueName: \"kubernetes.io/projected/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-api-access-4ltf2\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667488 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667349    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-data-volume\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.667488 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba42c95-d563-4e39-b778-6a76b43761e3-config-volume\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667689 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba42c95-d563-4e39-b778-6a76b43761e3-tmp-dir\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.667788 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.667707    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.668071 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.668042    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba42c95-d563-4e39-b778-6a76b43761e3-config-volume\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.671121 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.671095    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba42c95-d563-4e39-b778-6a76b43761e3-metrics-tls\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.671201 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.671141    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464150bb-e3fb-42f5-881e-a3e4f64b08f4-cert\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.674612 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.674589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.679256 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.679238    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltf2\" (UniqueName: \"kubernetes.io/projected/7ed6e45f-4729-4d51-9e0f-1b2c8013cd56-kube-api-access-4ltf2\") pod \"insights-runtime-extractor-ktjs7\" (UID: \"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56\") " pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.680320 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.680302    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsjf\" (UniqueName: \"kubernetes.io/projected/8ba42c95-d563-4e39-b778-6a76b43761e3-kube-api-access-wfsjf\") pod \"dns-default-fnjkr\" (UID: \"8ba42c95-d563-4e39-b778-6a76b43761e3\") " pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:38.680697 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.680681    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7jn\" (UniqueName: \"kubernetes.io/projected/464150bb-e3fb-42f5-881e-a3e4f64b08f4-kube-api-access-vd7jn\") pod \"ingress-canary-j6jx6\" (UID: \"464150bb-e3fb-42f5-881e-a3e4f64b08f4\") " pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.709220 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.709195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:38.806260 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.806213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:38.808121 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.808105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6jx6"
Jul 01 20:47:38.819310 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.819296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ktjs7"
Jul 01 20:47:38.851600 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:38.851580    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:39.004269 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.004241    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ktjs7"]
Jul 01 20:47:39.007771 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:39.007740    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6e45f_4729_4d51_9e0f_1b2c8013cd56.slice/crio-6790187caba5a4188826fcbc0d022fc470f7297e25e6d21a220b369a0647542b WatchSource:0}: Error finding container 6790187caba5a4188826fcbc0d022fc470f7297e25e6d21a220b369a0647542b: Status 404 returned error can't find the container with id 6790187caba5a4188826fcbc0d022fc470f7297e25e6d21a220b369a0647542b
Jul 01 20:47:39.008514 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.008492    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6jx6"]
Jul 01 20:47:39.011544 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:39.011522    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464150bb_e3fb_42f5_881e_a3e4f64b08f4.slice/crio-f8a7bda8392cc79bae89b0004b6e525d80d99e1b7d9409e8eca0dbb9952f3e42 WatchSource:0}: Error finding container f8a7bda8392cc79bae89b0004b6e525d80d99e1b7d9409e8eca0dbb9952f3e42: Status 404 returned error can't find the container with id f8a7bda8392cc79bae89b0004b6e525d80d99e1b7d9409e8eca0dbb9952f3e42
Jul 01 20:47:39.016535 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.016515    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnjkr"]
Jul 01 20:47:39.019423 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:39.019405    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba42c95_d563_4e39_b778_6a76b43761e3.slice/crio-2f3b137728d6e6d23925fd8cc00430b9a1eab9262da2d6c34aae1679323b90b6 WatchSource:0}: Error finding container 2f3b137728d6e6d23925fd8cc00430b9a1eab9262da2d6c34aae1679323b90b6: Status 404 returned error can't find the container with id 2f3b137728d6e6d23925fd8cc00430b9a1eab9262da2d6c34aae1679323b90b6
Jul 01 20:47:39.864097 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.863664    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnjkr" event={"ID":"8ba42c95-d563-4e39-b778-6a76b43761e3","Type":"ContainerStarted","Data":"2f3b137728d6e6d23925fd8cc00430b9a1eab9262da2d6c34aae1679323b90b6"}
Jul 01 20:47:39.865402 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.865318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktjs7" event={"ID":"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56","Type":"ContainerStarted","Data":"6790187caba5a4188826fcbc0d022fc470f7297e25e6d21a220b369a0647542b"}
Jul 01 20:47:39.867729 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:39.867692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6jx6" event={"ID":"464150bb-e3fb-42f5-881e-a3e4f64b08f4","Type":"ContainerStarted","Data":"f8a7bda8392cc79bae89b0004b6e525d80d99e1b7d9409e8eca0dbb9952f3e42"}
Jul 01 20:47:44.882739 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.882629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnjkr" event={"ID":"8ba42c95-d563-4e39-b778-6a76b43761e3","Type":"ContainerStarted","Data":"30147169bb49bd14fa1e797584a25676cde5a1b29c0534ca32c1ce51ac0cd539"}
Jul 01 20:47:44.883468 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.882755    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnjkr" event={"ID":"8ba42c95-d563-4e39-b778-6a76b43761e3","Type":"ContainerStarted","Data":"bf36343247d6f26b8659c877126a2d98f0fa95389e45f19a5cb7671cf698ddeb"}
Jul 01 20:47:44.883468 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.882787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:44.884075 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.884054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktjs7" event={"ID":"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56","Type":"ContainerStarted","Data":"df4b33356050eeee0ef0ef21deb4e00c6ddf7c651f8c23247aff07667a43da52"}
Jul 01 20:47:44.885407 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.885382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6jx6" event={"ID":"464150bb-e3fb-42f5-881e-a3e4f64b08f4","Type":"ContainerStarted","Data":"3e2936c38fd5872fcf4b806d39534dd5be97325373aeb32afa43ed47ee38a181"}
Jul 01 20:47:44.900818 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.900782    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fnjkr" podStartSLOduration=1.32382406 podStartE2EDuration="6.900772312s" totalImagesPullingTime="5.576948252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:44.900091846 +0000 UTC m=+55.815369913" watchObservedRunningTime="2026-07-01 20:47:44.900772312 +0000 UTC m=+55.816050353"
Jul 01 20:47:44.918408 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:44.918371    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6jx6" podStartSLOduration=1.3153188519999999 podStartE2EDuration="6.918359072s" totalImagesPullingTime="5.60304022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:44.917662818 +0000 UTC m=+55.832940856" watchObservedRunningTime="2026-07-01 20:47:44.918359072 +0000 UTC m=+55.833637110"
Jul 01 20:47:45.754283 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:45.754120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:45.811419 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:45.811386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:45.891642 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:45.891606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktjs7" event={"ID":"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56","Type":"ContainerStarted","Data":"a3517a0e3e86f4e50a889874fbd6ec114a1fc48bc8f622f540d21f002c195d59"}
Jul 01 20:47:46.414066 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:46.414031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:46.511622 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:46.511586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:46.592435 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:46.592399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:46.664937 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:46.664864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:46.984524 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:46.984446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:47.105654 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.105626    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:47.219059 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.219029    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:47:47.225324 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.225300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.231216 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.231193    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 20:47:47.231361 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.231192    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 20:47:47.232800 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.232782    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 20:47:47.233080 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.233056    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2b7cg\""
Jul 01 20:47:47.233262 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.233244    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 20:47:47.233443 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.233430    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 20:47:47.233631 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.233614    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 20:47:47.233836 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.233821    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 20:47:47.238087 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.238036    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:47:47.337776 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.337739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.337944 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.337781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h9x\" (UniqueName: \"kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.337944 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.337873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.338045 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.337934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.338045 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.337982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.338045 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.338033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439199 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439199 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439393 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439230    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439393 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439393 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.439587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.439403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h9x\" (UniqueName: \"kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.440416 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.440368    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.440550 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.440419    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.440744 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.440677    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.444759 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.444736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.444879 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.444737    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.448406 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.448382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h9x\" (UniqueName: \"kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x\") pod \"console-5d7c8949d7-fxjdz\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") " pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.537577 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.537498    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:47.763553 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.763530    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:47:47.766405 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:47.766377    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd55279_3776_47bf_ba09_eef5daa9bbcb.slice/crio-fccb632a9fe79a70869d5415f76190895321636997c78378a5c911076e7444e8 WatchSource:0}: Error finding container fccb632a9fe79a70869d5415f76190895321636997c78378a5c911076e7444e8: Status 404 returned error can't find the container with id fccb632a9fe79a70869d5415f76190895321636997c78378a5c911076e7444e8
Jul 01 20:47:47.901874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.901803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktjs7" event={"ID":"7ed6e45f-4729-4d51-9e0f-1b2c8013cd56","Type":"ContainerStarted","Data":"fd53d3d62f65d596de9860280cd09b38222b1b575a9703b596d4d3352280d28c"}
Jul 01 20:47:47.903107 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.903084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7c8949d7-fxjdz" event={"ID":"afd55279-3776-47bf-ba09-eef5daa9bbcb","Type":"ContainerStarted","Data":"fccb632a9fe79a70869d5415f76190895321636997c78378a5c911076e7444e8"}
Jul 01 20:47:47.920125 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:47.920089    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ktjs7" podStartSLOduration=1.404282334 podStartE2EDuration="9.920074464s" totalImagesPullingTime="8.51579213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:47.919104752 +0000 UTC m=+58.834382827" watchObservedRunningTime="2026-07-01 20:47:47.920074464 +0000 UTC m=+58.835352480"
Jul 01 20:47:49.624961 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.624929    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:49.664679 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.664650    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"]
Jul 01 20:47:49.666973 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.666947    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.669739 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.669702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 20:47:49.670660 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.670642    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 20:47:49.670795 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.670660    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 20:47:49.670795 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.670675    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 20:47:49.670795 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.670690    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 20:47:49.672192 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.672171    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z9kcx\""
Jul 01 20:47:49.682857 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.682839    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"]
Jul 01 20:47:49.723260 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.723235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:49.756861 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.756836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.756977 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.756863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd9l\" (UniqueName: \"kubernetes.io/projected/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-kube-api-access-npd9l\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.756977 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.756919    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.756977 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.756946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.858093 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.858061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.858225 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.858110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.858225 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.858165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.858225 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.858184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd9l\" (UniqueName: \"kubernetes.io/projected/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-kube-api-access-npd9l\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.858377 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:49.858333    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 01 20:47:49.858429 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:49.858399    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls podName:7ee3ab90-3ee4-4244-84f7-f34e66edf21f nodeName:}" failed. No retries permitted until 2026-07-01 20:47:50.358379875 +0000 UTC m=+61.273657905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-jlxhh" (UID: "7ee3ab90-3ee4-4244-84f7-f34e66edf21f") : secret "prometheus-operator-tls" not found
Jul 01 20:47:49.859156 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.859136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.860767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.860746    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:49.869530 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:49.869500    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd9l\" (UniqueName: \"kubernetes.io/projected/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-kube-api-access-npd9l\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:50.013428 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.013353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:50.083565 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.083516    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:50.162214 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.162185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:50.361925 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.361848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:50.364577 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.364546    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab90-3ee4-4244-84f7-f34e66edf21f-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-jlxhh\" (UID: \"7ee3ab90-3ee4-4244-84f7-f34e66edf21f\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:50.578059 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.578030    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"
Jul 01 20:47:50.783351 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.783321    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-jlxhh"]
Jul 01 20:47:50.787133 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:50.787106    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee3ab90_3ee4_4244_84f7_f34e66edf21f.slice/crio-d5ef8eca89d192182857ac4c67924a00fff667abec47c1e80cd1e8870cb31de6 WatchSource:0}: Error finding container d5ef8eca89d192182857ac4c67924a00fff667abec47c1e80cd1e8870cb31de6: Status 404 returned error can't find the container with id d5ef8eca89d192182857ac4c67924a00fff667abec47c1e80cd1e8870cb31de6
Jul 01 20:47:50.828291 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.828270    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7jjmd"
Jul 01 20:47:50.914807 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.914743    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh" event={"ID":"7ee3ab90-3ee4-4244-84f7-f34e66edf21f","Type":"ContainerStarted","Data":"d5ef8eca89d192182857ac4c67924a00fff667abec47c1e80cd1e8870cb31de6"}
Jul 01 20:47:50.963887 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:50.963828    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:51.035106 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:51.035072    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:51.104159 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:51.104128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:51.920261 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:51.920227    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7c8949d7-fxjdz" event={"ID":"afd55279-3776-47bf-ba09-eef5daa9bbcb","Type":"ContainerStarted","Data":"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"}
Jul 01 20:47:51.939561 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:51.939520    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d7c8949d7-fxjdz" podStartSLOduration=1.7610125779999999 podStartE2EDuration="4.93950502s" totalImagesPullingTime="3.178492442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:51.938773408 +0000 UTC m=+62.854051447" watchObservedRunningTime="2026-07-01 20:47:51.93950502 +0000 UTC m=+62.854783058"
Jul 01 20:47:51.969996 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:51.969974    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:52.070975 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:52.070919    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:52.332554 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:52.332521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:52.930247 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:52.930211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh" event={"ID":"7ee3ab90-3ee4-4244-84f7-f34e66edf21f","Type":"ContainerStarted","Data":"dca31d3c1db76d105b3ccef6fc4d0de66479258236a18f8a825a06c421521c05"}
Jul 01 20:47:52.930731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:52.930308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh" event={"ID":"7ee3ab90-3ee4-4244-84f7-f34e66edf21f","Type":"ContainerStarted","Data":"5aa3e29d127ca18e7dab3be8157d992698900812c2bf20222f6a9d4b372e5f70"}
Jul 01 20:47:52.948483 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:52.948438    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-jlxhh" podStartSLOduration=2.779390857 podStartE2EDuration="3.94842566s" totalImagesPullingTime="1.169034803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:52.947083916 +0000 UTC m=+63.862361953" watchObservedRunningTime="2026-07-01 20:47:52.94842566 +0000 UTC m=+63.863703724"
Jul 01 20:47:54.407826 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:54.407792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:54.508453 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:54.508397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:54.787869 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:54.787842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:54.893747 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:54.893703    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fnjkr"
Jul 01 20:47:55.095393 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.095328    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.119258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.119230    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"]
Jul 01 20:47:55.121449 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.121435    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.126213 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.126194    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-j2ts2\""
Jul 01 20:47:55.126457 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.126442    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 20:47:55.126545 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.126439    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 20:47:55.148695 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.148674    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"]
Jul 01 20:47:55.166919 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.166901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-pwgkr"]
Jul 01 20:47:55.169157 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.169139    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.169768 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.169748    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c9vrh"]
Jul 01 20:47:55.171625 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.171610    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.176631 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.176612    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 20:47:55.176858 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.176843    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kz88c\""
Jul 01 20:47:55.176941 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.176863    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 20:47:55.176941 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.176878    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 20:47:55.176941 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.176921    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bx4px\""
Jul 01 20:47:55.178315 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.178294    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 20:47:55.178315 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.178312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 20:47:55.178460 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.178314    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 20:47:55.190196 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.190177    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-pwgkr"]
Jul 01 20:47:55.198258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.198240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.198350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.198265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f6a7cde-81af-40d7-954f-cbdebb51075b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.198350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.198286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtcw\" (UniqueName: \"kubernetes.io/projected/8f6a7cde-81af-40d7-954f-cbdebb51075b-kube-api-access-lxtcw\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.198350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.198322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.232316 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.232292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.299376 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.299512 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-wtmp\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299512 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299512 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5deae4be-fe60-4f24-8415-da74dace7ff4-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.299512 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-metrics-client-ca\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-metrics-client-ca\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.299661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jkw\" (UniqueName: \"kubernetes.io/projected/c73f447e-3bdc-4af4-9cee-32591ebd060e-kube-api-access-r4jkw\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.299798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7zx\" (UniqueName: \"kubernetes.io/projected/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-api-access-wm7zx\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.299798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.299798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-textfile\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-root\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-sys\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f6a7cde-81af-40d7-954f-cbdebb51075b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtcw\" (UniqueName: \"kubernetes.io/projected/8f6a7cde-81af-40d7-954f-cbdebb51075b-kube-api-access-lxtcw\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.299982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.299954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.300605 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.300509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f6a7cde-81af-40d7-954f-cbdebb51075b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.302319 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.302296    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.302441 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.302353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6a7cde-81af-40d7-954f-cbdebb51075b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.302680 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.302668    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:55.310688 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.310669    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtcw\" (UniqueName: \"kubernetes.io/projected/8f6a7cde-81af-40d7-954f-cbdebb51075b-kube-api-access-lxtcw\") pod \"openshift-state-metrics-57f99c8b49-2kw5z\" (UID: \"8f6a7cde-81af-40d7-954f-cbdebb51075b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.312131 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.312106    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68e6e793-e805-4c65-8b94-5d540f3bf401-original-pull-secret\") pod \"global-pull-secret-syncer-77s9m\" (UID: \"68e6e793-e805-4c65-8b94-5d540f3bf401\") " pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:55.313752 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.313729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.362531 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.362465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77s9m"
Jul 01 20:47:55.400705 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7zx\" (UniqueName: \"kubernetes.io/projected/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-api-access-wm7zx\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.400874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:55.400874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-textfile\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.400874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.400874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-root\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.400874 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-sys\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.400979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-wtmp\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401043    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-sys\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5deae4be-fe60-4f24-8415-da74dace7ff4-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-root\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-metrics-client-ca\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401159    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-textfile\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401182    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-wtmp\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:55.401215    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-metrics-client-ca\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jkw\" (UniqueName: \"kubernetes.io/projected/c73f447e-3bdc-4af4-9cee-32591ebd060e-kube-api-access-r4jkw\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:47:55.401284    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls podName:c73f447e-3bdc-4af4-9cee-32591ebd060e nodeName:}" failed. No retries permitted until 2026-07-01 20:47:55.901265291 +0000 UTC m=+66.816543309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls") pod "node-exporter-c9vrh" (UID: "c73f447e-3bdc-4af4-9cee-32591ebd060e") : secret "node-exporter-tls" not found
Jul 01 20:47:55.401767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401650    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-metrics-client-ca\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.402104 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.401903    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.402104 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.402028    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5deae4be-fe60-4f24-8415-da74dace7ff4-metrics-client-ca\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.402104 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.402026    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5deae4be-fe60-4f24-8415-da74dace7ff4-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.402104 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.402059    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.403888 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.403864    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:55.403888 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.403885    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:47:55.404046 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.403950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.404109 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.404048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.404560 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.404543    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.413836 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.413809    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c08e116-b49a-4f87-a27c-2e4a5b0cfece-metrics-certs\") pod \"network-metrics-daemon-9jvqx\" (UID: \"4c08e116-b49a-4f87-a27c-2e4a5b0cfece\") " pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:55.414482 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.414463    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:47:55.424429 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.424402    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jkw\" (UniqueName: \"kubernetes.io/projected/c73f447e-3bdc-4af4-9cee-32591ebd060e-kube-api-access-r4jkw\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.424541 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.424408    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bc5\" (UniqueName: \"kubernetes.io/projected/2eb44aee-dfbc-45b7-9ee4-45ee9acbd117-kube-api-access-b9bc5\") pod \"network-check-target-8csxj\" (UID: \"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117\") " pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:55.426231 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.426212    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7zx\" (UniqueName: \"kubernetes.io/projected/5deae4be-fe60-4f24-8415-da74dace7ff4-kube-api-access-wm7zx\") pod \"kube-state-metrics-845b954455-pwgkr\" (UID: \"5deae4be-fe60-4f24-8415-da74dace7ff4\") " pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.429884 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.429865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"
Jul 01 20:47:55.439958 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.439936    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.478320 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.478293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr"
Jul 01 20:47:55.487790 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.487655    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77s9m"]
Jul 01 20:47:55.507971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.507946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.571466 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:55.571440    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6a7cde_81af_40d7_954f_cbdebb51075b.slice/crio-bb697cd4b9c126cee48382ee60d42d48d35df1e381327edb02f1c594c8ab75f7 WatchSource:0}: Error finding container bb697cd4b9c126cee48382ee60d42d48d35df1e381327edb02f1c594c8ab75f7: Status 404 returned error can't find the container with id bb697cd4b9c126cee48382ee60d42d48d35df1e381327edb02f1c594c8ab75f7
Jul 01 20:47:55.573046 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.573000    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:55.573195 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.573177    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z"]
Jul 01 20:47:55.613603 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.613559    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-pwgkr"]
Jul 01 20:47:55.616084 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:55.616055    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deae4be_fe60_4f24_8415_da74dace7ff4.slice/crio-6b7900146f8b421de0567369fece03b9e4add1bfabed4f6ca4404cc9d200f5c4 WatchSource:0}: Error finding container 6b7900146f8b421de0567369fece03b9e4add1bfabed4f6ca4404cc9d200f5c4: Status 404 returned error can't find the container with id 6b7900146f8b421de0567369fece03b9e4add1bfabed4f6ca4404cc9d200f5c4
Jul 01 20:47:55.658988 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.658943    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64glk\""
Jul 01 20:47:55.667320 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.667297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:47:55.669754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.669733    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h98zr\""
Jul 01 20:47:55.677595 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.677577    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jvqx"
Jul 01 20:47:55.802200 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.802173    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8csxj"]
Jul 01 20:47:55.804841 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:55.804819    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb44aee_dfbc_45b7_9ee4_45ee9acbd117.slice/crio-020f457ecfec3e47821445d0bf5ef40ec904334ad84dd19f6194c059d747e31d WatchSource:0}: Error finding container 020f457ecfec3e47821445d0bf5ef40ec904334ad84dd19f6194c059d747e31d: Status 404 returned error can't find the container with id 020f457ecfec3e47821445d0bf5ef40ec904334ad84dd19f6194c059d747e31d
Jul 01 20:47:55.826244 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.826224    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jvqx"]
Jul 01 20:47:55.829079 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:55.829059    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c08e116_b49a_4f87_a27c_2e4a5b0cfece.slice/crio-c8b2d906092fe88d591111b18f38fb2d6da5a35588c1787338782d8b26934360 WatchSource:0}: Error finding container c8b2d906092fe88d591111b18f38fb2d6da5a35588c1787338782d8b26934360: Status 404 returned error can't find the container with id c8b2d906092fe88d591111b18f38fb2d6da5a35588c1787338782d8b26934360
Jul 01 20:47:55.904038 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.903990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.906204 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.906184    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c73f447e-3bdc-4af4-9cee-32591ebd060e-node-exporter-tls\") pod \"node-exporter-c9vrh\" (UID: \"c73f447e-3bdc-4af4-9cee-32591ebd060e\") " pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:55.940849 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.940821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr" event={"ID":"5deae4be-fe60-4f24-8415-da74dace7ff4","Type":"ContainerStarted","Data":"6b7900146f8b421de0567369fece03b9e4add1bfabed4f6ca4404cc9d200f5c4"}
Jul 01 20:47:55.941996 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.941844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77s9m" event={"ID":"68e6e793-e805-4c65-8b94-5d540f3bf401","Type":"ContainerStarted","Data":"7f84a9486dba30b54b9e1e70057bcda10c6da4de1f99239e9a70bd6bbfd9df9b"}
Jul 01 20:47:55.942853 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.942832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8csxj" event={"ID":"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117","Type":"ContainerStarted","Data":"020f457ecfec3e47821445d0bf5ef40ec904334ad84dd19f6194c059d747e31d"}
Jul 01 20:47:55.943884 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.943868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jvqx" event={"ID":"4c08e116-b49a-4f87-a27c-2e4a5b0cfece","Type":"ContainerStarted","Data":"c8b2d906092fe88d591111b18f38fb2d6da5a35588c1787338782d8b26934360"}
Jul 01 20:47:55.945579 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.945562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z" event={"ID":"8f6a7cde-81af-40d7-954f-cbdebb51075b","Type":"ContainerStarted","Data":"b5cd758621f30cf07235358e063446e3b563de301a9505fbdab0fe62902684d3"}
Jul 01 20:47:55.945660 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.945597    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z" event={"ID":"8f6a7cde-81af-40d7-954f-cbdebb51075b","Type":"ContainerStarted","Data":"cb700e74106daa21ebe5489420ad8fca248a635dbe1aa13d5ddfbf9d3f6662b1"}
Jul 01 20:47:55.945660 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:55.945615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z" event={"ID":"8f6a7cde-81af-40d7-954f-cbdebb51075b","Type":"ContainerStarted","Data":"bb697cd4b9c126cee48382ee60d42d48d35df1e381327edb02f1c594c8ab75f7"}
Jul 01 20:47:56.016595 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.016573    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.083699 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.083674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9vrh"
Jul 01 20:47:56.094763 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:47:56.094738    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f447e_3bdc_4af4_9cee_32591ebd060e.slice/crio-45b630b777198b37c44baa99e2808801984efaefd5f58526897b362822e974a1 WatchSource:0}: Error finding container 45b630b777198b37c44baa99e2808801984efaefd5f58526897b362822e974a1: Status 404 returned error can't find the container with id 45b630b777198b37c44baa99e2808801984efaefd5f58526897b362822e974a1
Jul 01 20:47:56.119222 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.119190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.196500 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.196462    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.444073 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.444034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.516893 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.516797    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.687880 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.687848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:56.951751 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:56.951693    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9vrh" event={"ID":"c73f447e-3bdc-4af4-9cee-32591ebd060e","Type":"ContainerStarted","Data":"45b630b777198b37c44baa99e2808801984efaefd5f58526897b362822e974a1"}
Jul 01 20:47:57.030523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.030405    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:57.128619 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.128488    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:57.538431 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.538400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:57.538875 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.538465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:57.548439 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.548408    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:57.962162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:57.962134    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:47:58.002024 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.001992    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:58.091956 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.091917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:58.096043 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.096020    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"]
Jul 01 20:47:58.113939 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.113914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.115835 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.115810    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"]
Jul 01 20:47:58.116912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.116892    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 01 20:47:58.117036 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.117013    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6cg6w\""
Jul 01 20:47:58.117140 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.117119    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 01 20:47:58.117291 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.117274    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 01 20:47:58.117369 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.117304    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-51pb5aosnmo1d\""
Jul 01 20:47:58.117426 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.116892    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 01 20:47:58.117536 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.117494    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 01 20:47:58.226706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6cad7-b26a-40ae-b673-25d3a593a741-metrics-client-ca\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-grpc-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226706 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226980 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226980 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226980 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.226980 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wn9x\" (UniqueName: \"kubernetes.io/projected/26c6cad7-b26a-40ae-b673-25d3a593a741-kube-api-access-9wn9x\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.227183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.226978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.259288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.259259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:58.327582 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6cad7-b26a-40ae-b673-25d3a593a741-metrics-client-ca\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327582 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-grpc-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wn9x\" (UniqueName: \"kubernetes.io/projected/26c6cad7-b26a-40ae-b673-25d3a593a741-kube-api-access-9wn9x\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.327846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.327790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.328462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.328423    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6cad7-b26a-40ae-b673-25d3a593a741-metrics-client-ca\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.331359 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.331332    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.332101 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.331743    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-grpc-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.332101 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.332053    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.332202 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.332135    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.333456 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.333432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.334081 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.334032    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26c6cad7-b26a-40ae-b673-25d3a593a741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.336896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.336876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wn9x\" (UniqueName: \"kubernetes.io/projected/26c6cad7-b26a-40ae-b673-25d3a593a741-kube-api-access-9wn9x\") pod \"thanos-querier-6f87cd9f75-tb7n8\" (UID: \"26c6cad7-b26a-40ae-b673-25d3a593a741\") " pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:58.352226 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.352190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:58.425735 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:58.425513    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:47:59.798446 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:59.798386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:59.864148 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:59.864105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:47:59.936035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:47:59.936004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:00.037204 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:00.037134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:00.761697 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:00.761660    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:01.347876 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.347619    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"]
Jul 01 20:48:01.350298 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:48:01.350259    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c6cad7_b26a_40ae_b673_25d3a593a741.slice/crio-2f78258c343ab42f965fa2472ee217983c1cb1ece686a7918a271f249fda65d5 WatchSource:0}: Error finding container 2f78258c343ab42f965fa2472ee217983c1cb1ece686a7918a271f249fda65d5: Status 404 returned error can't find the container with id 2f78258c343ab42f965fa2472ee217983c1cb1ece686a7918a271f249fda65d5
Jul 01 20:48:01.467949 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.466578    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:48:01.475931 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.475803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:01.485617 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.483427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.485617 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.485531    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:48:01.490038 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490013    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 20:48:01.490783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kb79z\""
Jul 01 20:48:01.491055 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.491011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 20:48:01.491422 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490025    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.491680    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.491764    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490349    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490904    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.491574    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490327    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 20:48:01.492251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.490245    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 20:48:01.496731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.493006    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dcd6cnj12mnuf\""
Jul 01 20:48:01.496731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.493195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 20:48:01.496731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.496519    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 20:48:01.498126 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.498108    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 20:48:01.498888 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.498872    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 20:48:01.502882 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.501358    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvb27\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.572998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.573593 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.574496 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.574496 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.574496 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.574496 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.573352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674030 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.673943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674030 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.673978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674030 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674212    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvb27\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.674864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674778    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.675140 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.674867    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.676658 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.676009    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.676658 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.676364    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.677893 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.677869    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.678141 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.678116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.678373 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.678108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.678373 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.677872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.678373 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.678210    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.678550 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.678529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.681799 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.679219    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.681799 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.680501    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.681799 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.680680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.681799 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.681068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.681799 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.681421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.682078 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.681904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.682335 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.682317    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.683669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.683647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.683810 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.683794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.692009 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.691992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvb27\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27\") pod \"prometheus-k8s-0\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.820130 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.820085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:01.835756 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.835495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:01.968166 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.968132    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:48:01.971191 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:48:01.971163    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec3a07c_ebed_41bd_b0d5_eee28a097ae7.slice/crio-77e70a340d5444bfd71ef7578022bb8b76edc94ac18e7502ad8ec07bc6140d24 WatchSource:0}: Error finding container 77e70a340d5444bfd71ef7578022bb8b76edc94ac18e7502ad8ec07bc6140d24: Status 404 returned error can't find the container with id 77e70a340d5444bfd71ef7578022bb8b76edc94ac18e7502ad8ec07bc6140d24
Jul 01 20:48:01.976230 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.976183    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z" event={"ID":"8f6a7cde-81af-40d7-954f-cbdebb51075b","Type":"ContainerStarted","Data":"852ce5d69d1fe94a97fac56a046b47a737dca8204b675031fc65515671acc41c"}
Jul 01 20:48:01.978858 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.978834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr" event={"ID":"5deae4be-fe60-4f24-8415-da74dace7ff4","Type":"ContainerStarted","Data":"324e1f07a68543528276b20642e05da8f06f7a1fff63af99803c0250f1a8b6e1"}
Jul 01 20:48:01.978962 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.978889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr" event={"ID":"5deae4be-fe60-4f24-8415-da74dace7ff4","Type":"ContainerStarted","Data":"bb06b6648c88aa95240d96150a917667c39db2e59e00fea5d4ae293ce8984d03"}
Jul 01 20:48:01.978962 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.978921    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr" event={"ID":"5deae4be-fe60-4f24-8415-da74dace7ff4","Type":"ContainerStarted","Data":"1ac2b81555dfdecb297e0e752c553ca14931b6faa5beb99c8662936548a917c4"}
Jul 01 20:48:01.980529 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.980476    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77s9m" event={"ID":"68e6e793-e805-4c65-8b94-5d540f3bf401","Type":"ContainerStarted","Data":"97504cc07aa1247a5db0e3550e2f6491d21d4713756f7207a8a00e94a8acecbd"}
Jul 01 20:48:01.984742 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.982576    2574 generic.go:356] "Generic (PLEG): container finished" podID="c73f447e-3bdc-4af4-9cee-32591ebd060e" containerID="c887104e6e8eda1f56fe1f42a5d2f3b1464c15dfbd953c54f0c538b7d76b614f" exitCode=0
Jul 01 20:48:01.984742 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.982657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9vrh" event={"ID":"c73f447e-3bdc-4af4-9cee-32591ebd060e","Type":"ContainerDied","Data":"c887104e6e8eda1f56fe1f42a5d2f3b1464c15dfbd953c54f0c538b7d76b614f"}
Jul 01 20:48:01.985212 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.985188    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"2f78258c343ab42f965fa2472ee217983c1cb1ece686a7918a271f249fda65d5"}
Jul 01 20:48:01.988946 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.987007    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8csxj" event={"ID":"2eb44aee-dfbc-45b7-9ee4-45ee9acbd117","Type":"ContainerStarted","Data":"a3d0faa8fd7ffca383a091f7b25adf6bdf697a0a43b661b8fb3f08ef0e8b9c89"}
Jul 01 20:48:01.988946 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.987062    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:48:01.990493 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.990471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jvqx" event={"ID":"4c08e116-b49a-4f87-a27c-2e4a5b0cfece","Type":"ContainerStarted","Data":"33d2a3b768f1f513f4de2b9e2e810ecf21078b60a165d843aeca22b2979a878c"}
Jul 01 20:48:01.990587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.990518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jvqx" event={"ID":"4c08e116-b49a-4f87-a27c-2e4a5b0cfece","Type":"ContainerStarted","Data":"89777b7d7a9721d0819f8cec68cf77c77a1c1a24d9ac62f82a87f1f3fa87ea6c"}
Jul 01 20:48:01.997171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:01.997121    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2kw5z" podStartSLOduration=1.52181084 podStartE2EDuration="6.997105342s" totalImagesPullingTime="5.475294502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:01.996853404 +0000 UTC m=+72.912131442" watchObservedRunningTime="2026-07-01 20:48:01.997105342 +0000 UTC m=+72.912383380"
Jul 01 20:48:02.012334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.012287    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-77s9m" podStartSLOduration=66.291309971 podStartE2EDuration="1m12.012274136s" totalImagesPullingTime="5.720964165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:02.011910308 +0000 UTC m=+72.927188346" watchObservedRunningTime="2026-07-01 20:48:02.012274136 +0000 UTC m=+72.927552152"
Jul 01 20:48:02.031637 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.031585    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9jvqx" podStartSLOduration=66.710903788 podStartE2EDuration="1m12.031566242s" totalImagesPullingTime="5.320662454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:02.029435771 +0000 UTC m=+72.944713822" watchObservedRunningTime="2026-07-01 20:48:02.031566242 +0000 UTC m=+72.946844282"
Jul 01 20:48:02.052782 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.052733    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-pwgkr" podStartSLOduration=1.5177705179999998 podStartE2EDuration="7.052690576s" totalImagesPullingTime="5.534920058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:02.050860992 +0000 UTC m=+72.966139028" watchObservedRunningTime="2026-07-01 20:48:02.052690576 +0000 UTC m=+72.967968615"
Jul 01 20:48:02.088342 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.088291    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8csxj" podStartSLOduration=66.706226618 podStartE2EDuration="1m12.088273088s" totalImagesPullingTime="5.38204647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:02.087890937 +0000 UTC m=+73.003168988" watchObservedRunningTime="2026-07-01 20:48:02.088273088 +0000 UTC m=+73.003551124"
Jul 01 20:48:02.137074 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.137013    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:02.235332 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.235257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:02.997753 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.997696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9vrh" event={"ID":"c73f447e-3bdc-4af4-9cee-32591ebd060e","Type":"ContainerStarted","Data":"177d613e0c533e85f1c274d8efac8797e3f4ba59c807c776417a971aa557a132"}
Jul 01 20:48:02.998183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.997817    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9vrh" event={"ID":"c73f447e-3bdc-4af4-9cee-32591ebd060e","Type":"ContainerStarted","Data":"5a1e46775f47781c256103cd237773febf96de8ad2f31624b14194ba672671b6"}
Jul 01 20:48:03.000312 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:02.999111    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"77e70a340d5444bfd71ef7578022bb8b76edc94ac18e7502ad8ec07bc6140d24"}
Jul 01 20:48:03.016423 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:03.016370    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c9vrh" podStartSLOduration=2.7628739810000003 podStartE2EDuration="8.016351757s" totalImagesPullingTime="5.054633776s" totalInitContainerRuntime="198.844ms" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:03.015253423 +0000 UTC m=+73.930531460" watchObservedRunningTime="2026-07-01 20:48:03.016351757 +0000 UTC m=+73.931629796"
Jul 01 20:48:04.004994 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.004959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"cce3c03b05290e7aedb8fa053478cec1d040e0d9a2d8ba6d4960441fdff8f71a"}
Jul 01 20:48:04.005377 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.005033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"fc6a4112b1db268dbf15e1a55fba7c72de1e92e3360946520c14b3b08488adb6"}
Jul 01 20:48:04.005377 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.005061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"451e873cd17b07ca659d96f68f77b02b58c1c4097c6073147e46463a3a1f7191"}
Jul 01 20:48:04.006455 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.006433    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527" exitCode=0
Jul 01 20:48:04.006534 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.006470    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"}
Jul 01 20:48:04.511009 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.510968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:04.587907 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:04.587864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:05.014705 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:05.014669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"aa705b0daf6cb223520f60a4935d6bfac6217fb5817c5d379669116b4518df2e"}
Jul 01 20:48:05.015143 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:05.014808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"713f215454bef5e5bec194d943207b166b61a61dfbaee5a1095d04a3f7756f3a"}
Jul 01 20:48:05.015143 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:05.014852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" event={"ID":"26c6cad7-b26a-40ae-b673-25d3a593a741","Type":"ContainerStarted","Data":"a1f65c064709ef1b8f05016393638df180b8cd52ffd7aa81ba4441874192c3a7"}
Jul 01 20:48:05.015143 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:05.014893    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:48:05.041523 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:05.041437    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8" podStartSLOduration=4.006468333 podStartE2EDuration="7.041422313s" totalImagesPullingTime="3.03495398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:05.039893114 +0000 UTC m=+75.955171164" watchObservedRunningTime="2026-07-01 20:48:05.041422313 +0000 UTC m=+75.956700351"
Jul 01 20:48:07.127632 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:07.127572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:07.227252 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:07.227219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:08.031086 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"}
Jul 01 20:48:08.031265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"}
Jul 01 20:48:08.031265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"}
Jul 01 20:48:08.031265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031175    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"}
Jul 01 20:48:08.031265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"}
Jul 01 20:48:08.031265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.031224    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerStarted","Data":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"}
Jul 01 20:48:08.084035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.083969    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.110121648 podStartE2EDuration="7.083948058s" totalImagesPullingTime="4.949873135s" totalInitContainerRuntime="23.953275ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:48:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:08.079099001 +0000 UTC m=+78.994377084" watchObservedRunningTime="2026-07-01 20:48:08.083948058 +0000 UTC m=+78.999226108"
Jul 01 20:48:08.969518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:08.969486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:09.069149 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:09.069086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:10.613851 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:10.613747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:11.024864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:11.024838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f87cd9f75-tb7n8"
Jul 01 20:48:11.820580 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:11.820548    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:13.029474 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:13.029444    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:48:13.605419 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:13.605391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:13.702294 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:13.702269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:13.992025 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:13.991998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:14.095690 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:14.095660    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:14.405838 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:14.405766    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:14.474392 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:14.474368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:16.600448 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:16.600411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:16.702239 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:16.702214    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:31.610598 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:31.610566    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:31.677802 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:31.677773    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:48:33.002494 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:33.002465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8csxj"
Jul 01 20:48:38.057588 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.057520    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d7c8949d7-fxjdz" podUID="afd55279-3776-47bf-ba09-eef5daa9bbcb" containerName="console" containerID="cri-o://63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2" gracePeriod=15
Jul 01 20:48:38.323600 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.323544    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:48:38.349173 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349151    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.349312 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349194    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.349312 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.349312 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349265    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5h9x\" (UniqueName: \"kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.349613 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349588    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config" (OuterVolumeSpecName: "console-config") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:38.349846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349824    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.350123 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.349887    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert\") pod \"afd55279-3776-47bf-ba09-eef5daa9bbcb\" (UID: \"afd55279-3776-47bf-ba09-eef5daa9bbcb\") "
Jul 01 20:48:38.350123 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.350087    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:38.350348 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.350320    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:38.350398 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.350379    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca" (OuterVolumeSpecName: "service-ca") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:38.351672 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.351649    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:38.351833 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.351806    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x" (OuterVolumeSpecName: "kube-api-access-d5h9x") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "kube-api-access-d5h9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:38.351881 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.351865    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "afd55279-3776-47bf-ba09-eef5daa9bbcb" (UID: "afd55279-3776-47bf-ba09-eef5daa9bbcb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:38.451335 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.451317    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5h9x\" (UniqueName: \"kubernetes.io/projected/afd55279-3776-47bf-ba09-eef5daa9bbcb-kube-api-access-d5h9x\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:38.451335 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.451336    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-service-ca\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:38.451469 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.451346    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afd55279-3776-47bf-ba09-eef5daa9bbcb-oauth-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:38.451469 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.451355    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:38.451469 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:38.451365    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afd55279-3776-47bf-ba09-eef5daa9bbcb-console-oauth-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:48:39.147334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.147301    2574 generic.go:356] "Generic (PLEG): container finished" podID="afd55279-3776-47bf-ba09-eef5daa9bbcb" containerID="63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2" exitCode=0
Jul 01 20:48:39.147783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.147345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7c8949d7-fxjdz"
Jul 01 20:48:39.147783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.147384    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7c8949d7-fxjdz" event={"ID":"afd55279-3776-47bf-ba09-eef5daa9bbcb","Type":"ContainerDied","Data":"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"}
Jul 01 20:48:39.147783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.147436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7c8949d7-fxjdz" event={"ID":"afd55279-3776-47bf-ba09-eef5daa9bbcb","Type":"ContainerDied","Data":"fccb632a9fe79a70869d5415f76190895321636997c78378a5c911076e7444e8"}
Jul 01 20:48:39.147783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.147462    2574 scope.go:117] "RemoveContainer" containerID="63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"
Jul 01 20:48:39.156160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.156143    2574 scope.go:117] "RemoveContainer" containerID="63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"
Jul 01 20:48:39.156421 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:48:39.156399    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2\": container with ID starting with 63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2 not found: ID does not exist" containerID="63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"
Jul 01 20:48:39.156487 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.156428    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2"} err="failed to get container status \"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2\": rpc error: code = NotFound desc = could not find container \"63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2\": container with ID starting with 63b1b1c106fdb5953bc6811e928416d90bf30bf323613e98083201dd014db9a2 not found: ID does not exist"
Jul 01 20:48:39.168428 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.168407    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:48:39.171971 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.171953    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d7c8949d7-fxjdz"]
Jul 01 20:48:39.548028 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:48:39.547999    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd55279-3776-47bf-ba09-eef5daa9bbcb" path="/var/lib/kubelet/pods/afd55279-3776-47bf-ba09-eef5daa9bbcb/volumes"
Jul 01 20:49:01.820757 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:01.820707    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:01.840405 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:01.840379    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:01.853884 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:01.853860    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:01.914409 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:01.914381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:02.258880 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:02.258857    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:14.398699 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:14.398645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:14.599584 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:14.599556    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:19.820110 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.820074    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:19.821057 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821025    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="prometheus" containerID="cri-o://c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0" gracePeriod=600
Jul 01 20:49:19.821374 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821322    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="thanos-sidecar" containerID="cri-o://5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc" gracePeriod=600
Jul 01 20:49:19.821573 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821319    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-web" containerID="cri-o://ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df" gracePeriod=600
Jul 01 20:49:19.821669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821336    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="config-reloader" containerID="cri-o://fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b" gracePeriod=600
Jul 01 20:49:19.821669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821336    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy" containerID="cri-o://ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1" gracePeriod=600
Jul 01 20:49:19.821798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:19.821201    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3" gracePeriod=600
Jul 01 20:49:20.066733 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.066687    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.080196 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080115    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080196 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080171    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080390 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080202    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080390 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080251    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080458    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvb27\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080497 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080486    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080644 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080521    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080644 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080547    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.080644 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080557    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.080644 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.080574    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081369 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081345    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.081460 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081394    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081460 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081425    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081570 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081460    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081570 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081570 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081512    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081570 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081538    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081569    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081615    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081640    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081677    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs\") pod \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\" (UID: \"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7\") "
Jul 01 20:49:20.081982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081859    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.081982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081914    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.081982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081932    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.081982 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.081947    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-metrics-client-ca\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.083991 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.083788    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.083991 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.083862    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.083991 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.083911    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out" (OuterVolumeSpecName: "config-out") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:20.083991 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.083950    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:20.083991 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.083973    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.085746 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.084366    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.085746 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.084739    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.085746 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.084756    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:20.087697 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.087667    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27" (OuterVolumeSpecName: "kube-api-access-cvb27") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "kube-api-access-cvb27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:20.088021 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.087995    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.088356 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.088323    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.088452 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.088431    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config" (OuterVolumeSpecName: "config") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.088509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.088455    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.088586 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.088566    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.089495 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.089462    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.090134 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.090118    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:20.098064 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.098035    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config" (OuterVolumeSpecName: "web-config") pod "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" (UID: "5ec3a07c-ebed-41bd-b0d5-eee28a097ae7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:20.182568 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182534    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182568 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182562    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182568 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182573    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-metrics-client-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182582    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182594    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config-out\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182602    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182611    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvb27\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-kube-api-access-cvb27\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182621    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182629    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-db\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182639    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-web-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182646    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-kube-rbac-proxy\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182654    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-grpc-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182662    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182672    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182680    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182687    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-tls-assets\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.182816 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.182696    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:49:20.310513 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310478    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3" exitCode=0
Jul 01 20:49:20.310513 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310509    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1" exitCode=0
Jul 01 20:49:20.310513 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310519    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df" exitCode=0
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310528    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc" exitCode=0
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310536    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b" exitCode=0
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310544    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0" exitCode=0
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"}
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310573    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310640    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"}
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"}
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"}
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310729    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.310754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"}
Jul 01 20:49:20.311036 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"}
Jul 01 20:49:20.311036 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.310794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ec3a07c-ebed-41bd-b0d5-eee28a097ae7","Type":"ContainerDied","Data":"77e70a340d5444bfd71ef7578022bb8b76edc94ac18e7502ad8ec07bc6140d24"}
Jul 01 20:49:20.318791 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.318609    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.325504 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.325490    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.331970 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.331951    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.335041 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.335021    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:20.338681 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.338664    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.345168 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.345156    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.348289 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.348267    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:20.352365 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.352352    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.358621 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.358607    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.358871 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.358854    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.358923 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.358877    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.358923 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.358895    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.359149 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.359132    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.359191 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359154    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.359191 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359170    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.359409 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.359393    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.359450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359415    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.359450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359434    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.359658 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.359640    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.359705 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359661    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.359705 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359675    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.359913 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.359898    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.359952 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359919    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.359952 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.359932    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.360179 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.360160    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.360230 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360183    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.360230 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360198    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.360426 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:49:20.360412    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.360472 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360430    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.360472 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360442    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.360663 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360648    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.360731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360664    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.360875 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360858    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.360923 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.360875    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.361084 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361068    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.361142 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361084    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.361342 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361324    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.361396 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361343    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.361553 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361536    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.361597 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361554    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.361788 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361764    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.361836 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361788    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.361978 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361962    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.362020 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.361978    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.362137 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362119    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.362186 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362140    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.362342 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362322    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.362396 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362343    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.362556 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362537    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.362595 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362557    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.362771 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362753    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.362771 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362771    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.362967 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362951    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.363011 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.362968    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.363175 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363156    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.363213 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363175    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.363365 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363350    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.363404 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363365    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.363575 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363557    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.363618 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363575    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.363752 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363737    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.363803 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363753    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.363922 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363906    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.363960 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.363923    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.364103 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364088    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.364153 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364103    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.364288 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364270    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.364328 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364290    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.364451 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364436    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.364495 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364453    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.364662 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364647    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.364728 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364661    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.364878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364863    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.364915 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.364878    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.365048 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365032    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.365086 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365048    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.365225 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365210    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.365265 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365225    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.365396 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365379    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.365434 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365396    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.365579 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365562    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.365621 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365581    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.365783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365767    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.365826 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365783    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.365990 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365974    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.366023 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.365992    2574 scope.go:117] "RemoveContainer" containerID="0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"
Jul 01 20:49:20.366203 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366187    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3"} err="failed to get container status \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": rpc error: code = NotFound desc = could not find container \"0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3\": container with ID starting with 0bc439e4f5c659512fcdb38af571d659e52787186db60e97240b678b831e5de3 not found: ID does not exist"
Jul 01 20:49:20.366249 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366203    2574 scope.go:117] "RemoveContainer" containerID="ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"
Jul 01 20:49:20.366410 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366395    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1"} err="failed to get container status \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": rpc error: code = NotFound desc = could not find container \"ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1\": container with ID starting with ec9b0688b36a4f448cba2a3746b7fd54cf680a65e9202684c4f6b17ae1d128d1 not found: ID does not exist"
Jul 01 20:49:20.366454 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366409    2574 scope.go:117] "RemoveContainer" containerID="ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"
Jul 01 20:49:20.366609 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366594    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df"} err="failed to get container status \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": rpc error: code = NotFound desc = could not find container \"ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df\": container with ID starting with ed1df5229ecf165ad40773d3fafec2248aa2a744e3d30c71b1d4e1e8991d52df not found: ID does not exist"
Jul 01 20:49:20.366647 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366609    2574 scope.go:117] "RemoveContainer" containerID="5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"
Jul 01 20:49:20.366886 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366869    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc"} err="failed to get container status \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": rpc error: code = NotFound desc = could not find container \"5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc\": container with ID starting with 5116b938ee5f4880b968b07141cd939788d9479d45362fbf4d275f0a42bd06dc not found: ID does not exist"
Jul 01 20:49:20.366886 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.366885    2574 scope.go:117] "RemoveContainer" containerID="fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"
Jul 01 20:49:20.367086 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.367069    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b"} err="failed to get container status \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": rpc error: code = NotFound desc = could not find container \"fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b\": container with ID starting with fbc895d6d4828de44838b188420f06cd35c28290ea63e1d319d16c0202d0482b not found: ID does not exist"
Jul 01 20:49:20.367130 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.367089    2574 scope.go:117] "RemoveContainer" containerID="c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"
Jul 01 20:49:20.367284 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.367265    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0"} err="failed to get container status \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": rpc error: code = NotFound desc = could not find container \"c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0\": container with ID starting with c25a301bfde4cbc2c0ee440c42a4ff97751100426c86dc2cb0a7fcafb0d786c0 not found: ID does not exist"
Jul 01 20:49:20.367337 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.367286    2574 scope.go:117] "RemoveContainer" containerID="84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"
Jul 01 20:49:20.367509 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.367483    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527"} err="failed to get container status \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": rpc error: code = NotFound desc = could not find container \"84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527\": container with ID starting with 84f9603640ae9dbd3e6f523091bc366be4ce2c6d49a368a35df3b8b28f873527 not found: ID does not exist"
Jul 01 20:49:20.377152 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377129    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:20.377377 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377365    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-web"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377378    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-web"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377388    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="config-reloader"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377393    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="config-reloader"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377403    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="prometheus"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377409    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="prometheus"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377415    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy"
Jul 01 20:49:20.377418 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377420    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377426    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="init-config-reloader"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377431    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="init-config-reloader"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377437    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="thanos-sidecar"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377441    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="thanos-sidecar"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377447    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377453    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377460    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd55279-3776-47bf-ba09-eef5daa9bbcb" containerName="console"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377465    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd55279-3776-47bf-ba09-eef5daa9bbcb" containerName="console"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377503    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="thanos-sidecar"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377509    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377514    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-web"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377520    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="prometheus"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="config-reloader"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377530    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:20.377610 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.377535    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd55279-3776-47bf-ba09-eef5daa9bbcb" containerName="console"
Jul 01 20:49:20.381410 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.381392    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.384282 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384251    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 20:49:20.384282 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384270    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 20:49:20.384462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dcd6cnj12mnuf\""
Jul 01 20:49:20.384462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384251    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 20:49:20.384462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384260    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 20:49:20.384462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384420    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 20:49:20.384681 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384482    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 20:49:20.384768 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kb79z\""
Jul 01 20:49:20.384768 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 20:49:20.384920 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384800    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 20:49:20.384920 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.384854    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 20:49:20.385248 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.385232    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 20:49:20.385403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.385379    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 20:49:20.385491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.385445    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 20:49:20.385491 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.385447    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 20:49:20.388668 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.388650    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 20:49:20.391194 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.391179    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 20:49:20.399170 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.399143    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:20.485168 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485313 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977lp\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-kube-api-access-977lp\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485525 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485602    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.485789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.485674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586696 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586696 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977lp\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-kube-api-access-977lp\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586696 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.586969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.586994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.587363 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.587964    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.588112    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.588825    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.589271    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589587 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.589485    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.589935 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.589612    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.590219 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.590177    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.590372 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.590346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.590533 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.590509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.591062 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.590979    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.591062 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.590994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.591569 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.591548    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.592161 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.592139    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.592416 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.592389    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.592514 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.592452    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.592770 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.592750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.593071 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.593048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.593159 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.593144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.593459 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.593441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.595124 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.595109    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977lp\" (UniqueName: \"kubernetes.io/projected/ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb-kube-api-access-977lp\") pod \"prometheus-k8s-0\" (UID: \"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.690657 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.690618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:20.836330 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.836299    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:20.839271 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:49:20.839238    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce07ba80_c7de_47d5_895e_2f0a1b6b2cdb.slice/crio-7aa5cbc89c11c8985991472c990453fcafee73c36f8fbf1f3e52dd61aacb300f WatchSource:0}: Error finding container 7aa5cbc89c11c8985991472c990453fcafee73c36f8fbf1f3e52dd61aacb300f: Status 404 returned error can't find the container with id 7aa5cbc89c11c8985991472c990453fcafee73c36f8fbf1f3e52dd61aacb300f
Jul 01 20:49:20.916950 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:20.916924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:21.315856 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:21.315817    2574 generic.go:356] "Generic (PLEG): container finished" podID="ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb" containerID="e6702818451d7361772c0a81b063c9ec2cc3080d7011ba51bd2c4e20bb27009e" exitCode=0
Jul 01 20:49:21.316006 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:21.315899    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerDied","Data":"e6702818451d7361772c0a81b063c9ec2cc3080d7011ba51bd2c4e20bb27009e"}
Jul 01 20:49:21.316006 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:21.315988    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"7aa5cbc89c11c8985991472c990453fcafee73c36f8fbf1f3e52dd61aacb300f"}
Jul 01 20:49:21.553851 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:21.553818    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec3a07c-ebed-41bd-b0d5-eee28a097ae7" path="/var/lib/kubelet/pods/5ec3a07c-ebed-41bd-b0d5-eee28a097ae7/volumes"
Jul 01 20:49:22.324295 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"f7452d51610f02358680913c07336c0d4764fb55f63f003a7513faca092be22e"}
Jul 01 20:49:22.324655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"f673fcdad41b7bff0d1c063a12bb7b060da7f883231d83ff5dea1b1283636c19"}
Jul 01 20:49:22.324655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"6b15904034d6c799fd829764243abef6fb1d5958aeb30664e38f4d4269481709"}
Jul 01 20:49:22.324655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324379    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"d6456c347d5ecf56e7f98d8ddc0fe653f74cd7ed8dbda98a1dd4ef23a958fec7"}
Jul 01 20:49:22.324655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324401    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"060d829a3fa9a08bf42df75847f267851a2fa754136943fc6e0a26250e01a4ed"}
Jul 01 20:49:22.324655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.324422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb","Type":"ContainerStarted","Data":"f05a9dbcae5516cdecc78cca4e4b98fc8f3a5505af777485e1daddc873216aa5"}
Jul 01 20:49:22.355826 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:22.355779    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.336976638 podStartE2EDuration="2.355762656s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.786018ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:49:22.353893456 +0000 UTC m=+153.269171493" watchObservedRunningTime="2026-07-01 20:49:22.355762656 +0000 UTC m=+153.271040694"
Jul 01 20:49:23.282864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:23.282826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:23.344315 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:23.344283    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:23.419658 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:23.419634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:23.477815 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:23.477789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:24.029089 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:24.029055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:24.093436 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:24.093409    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:25.690930 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:25.690899    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:34.449643 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:34.449602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:34.550853 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:34.550826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:34.650921 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:34.650891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:49:53.823705 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:49:53.823668    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:05.643469 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:05.643434    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:05.705458 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:05.705425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:20.691228 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:20.691184    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:20.706969 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:20.706944    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:21.559932 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:21.559905    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:29.797966 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:29.797935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:29.898231 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:29.898195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:31.177683 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:31.177648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:31.279533 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:31.279503    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:41.660201 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:41.660163    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:41.783283 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:41.783245    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:41.867324 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:41.867286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:46.640556 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:46.640526    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:46.742867 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:46.742833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:50:58.087450 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:50:58.087404    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:03.205002 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:03.204956    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:03.267452 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:03.267424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:20.145949 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:20.145898    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:20.238096 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:20.238066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:20.430805 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:20.430705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:20.495408 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:20.495381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:20.565600 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:20.565571    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:42.960772 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:42.960730    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:44.280402 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:44.280362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:49.491376 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:49.491338    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:51:49.494784 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:49.494758    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 20:51:49.501542 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:49.501524    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:51:51.274058 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:51.274027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:51.716479 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:51.716433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:51.788883 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:51.788850    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.028487 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.028412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.125348 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.125298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.226298 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.226262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.426073 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.426001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.625609 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.625575    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.884962 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.884931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:51:52.954731 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:51:52.954690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:00.296283 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.296199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:00.417624 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.417594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:00.424765 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.424741    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-krbb5"]
Jul 01 20:53:00.426791 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.426777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.429967 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.429948    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 20:53:00.430726 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.430696    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pd6gz\""
Jul 01 20:53:00.430792 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.430762    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 20:53:00.430792 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.430781    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 20:53:00.440942 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.440915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-krbb5"]
Jul 01 20:53:00.464300 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.464280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-data\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.464381 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.464315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcw6\" (UniqueName: \"kubernetes.io/projected/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-kube-api-access-6dcw6\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.550307 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.550247    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:00.565438 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.565417    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-data\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.565517 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.565448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcw6\" (UniqueName: \"kubernetes.io/projected/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-kube-api-access-6dcw6\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.565800 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.565782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-data\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.592879 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.592850    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcw6\" (UniqueName: \"kubernetes.io/projected/eab2b0a9-e895-4e31-befe-e007b8ffa0a6-kube-api-access-6dcw6\") pod \"seaweedfs-64568bcd49-krbb5\" (UID: \"eab2b0a9-e895-4e31-befe-e007b8ffa0a6\") " pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.735864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.735835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:00.755846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.755817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:00.857539 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.857404    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-krbb5"]
Jul 01 20:53:00.860060 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:53:00.860018    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab2b0a9_e895_4e31_befe_e007b8ffa0a6.slice/crio-dc8e98e74c9da96ecf8c8d5100e8976c04a3b63753b607686b3c87f9757f6613 WatchSource:0}: Error finding container dc8e98e74c9da96ecf8c8d5100e8976c04a3b63753b607686b3c87f9757f6613: Status 404 returned error can't find the container with id dc8e98e74c9da96ecf8c8d5100e8976c04a3b63753b607686b3c87f9757f6613
Jul 01 20:53:00.861344 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:00.861330    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:53:01.091769 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:01.091690    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-krbb5" event={"ID":"eab2b0a9-e895-4e31-befe-e007b8ffa0a6","Type":"ContainerStarted","Data":"dc8e98e74c9da96ecf8c8d5100e8976c04a3b63753b607686b3c87f9757f6613"}
Jul 01 20:53:01.217911 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:01.217879    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:01.319323 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:01.319293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:01.419467 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:01.419397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:53:04.107972 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:04.107930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-krbb5" event={"ID":"eab2b0a9-e895-4e31-befe-e007b8ffa0a6","Type":"ContainerStarted","Data":"0730821c8d735e5e7c018180e94238757725a768465042c2ed95caf779d48eae"}
Jul 01 20:53:04.108391 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:04.108075    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:53:04.124760 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:04.124690    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-krbb5" podStartSLOduration=0.984786775 podStartE2EDuration="4.124674345s" totalImagesPullingTime="3.13988757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:53:04.123179693 +0000 UTC m=+375.038457731" watchObservedRunningTime="2026-07-01 20:53:04.124674345 +0000 UTC m=+375.039952382"
Jul 01 20:53:10.113699 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:53:10.113670    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-krbb5"
Jul 01 20:54:11.181207 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.181129    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.227044 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.227010    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-cwvfk"]
Jul 01 20:54:11.230054 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.230035    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.233613 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.233592    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 20:54:11.236447 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.236164    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6h4mt\""
Jul 01 20:54:11.239862 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.239844    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-cwvfk"]
Jul 01 20:54:11.255730 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.255690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.302768 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.302745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.302855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.302788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8f2\" (UniqueName: \"kubernetes.io/projected/b06fdbbf-8b20-44bc-9924-b03cf338c82e-kube-api-access-js8f2\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.403812 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.403791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.403914 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.403828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8f2\" (UniqueName: \"kubernetes.io/projected/b06fdbbf-8b20-44bc-9924-b03cf338c82e-kube-api-access-js8f2\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.403974 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:54:11.403941    2574 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 01 20:54:11.404026 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:54:11.404019    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs podName:b06fdbbf-8b20-44bc-9924-b03cf338c82e nodeName:}" failed. No retries permitted until 2026-07-01 20:54:11.903997809 +0000 UTC m=+442.819276025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs") pod "model-serving-api-fd65d7d6-cwvfk" (UID: "b06fdbbf-8b20-44bc-9924-b03cf338c82e") : secret "model-serving-api-tls" not found
Jul 01 20:54:11.415675 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.415654    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8f2\" (UniqueName: \"kubernetes.io/projected/b06fdbbf-8b20-44bc-9924-b03cf338c82e-kube-api-access-js8f2\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.521467 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.521444    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.623765 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.623743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.817499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.817433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.874502 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.874480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:11.907675 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.907648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:11.910093 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:11.910069    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06fdbbf-8b20-44bc-9924-b03cf338c82e-tls-certs\") pod \"model-serving-api-fd65d7d6-cwvfk\" (UID: \"b06fdbbf-8b20-44bc-9924-b03cf338c82e\") " pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:12.140783 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:12.140684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:12.266534 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:12.266502    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-cwvfk"]
Jul 01 20:54:12.269743 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:54:12.269686    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06fdbbf_8b20_44bc_9924_b03cf338c82e.slice/crio-6dd6179350ad90fd20dd503e1e514e6765967921be4eccedab3fea290c193370 WatchSource:0}: Error finding container 6dd6179350ad90fd20dd503e1e514e6765967921be4eccedab3fea290c193370: Status 404 returned error can't find the container with id 6dd6179350ad90fd20dd503e1e514e6765967921be4eccedab3fea290c193370
Jul 01 20:54:12.347393 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:12.347368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-cwvfk" event={"ID":"b06fdbbf-8b20-44bc-9924-b03cf338c82e","Type":"ContainerStarted","Data":"6dd6179350ad90fd20dd503e1e514e6765967921be4eccedab3fea290c193370"}
Jul 01 20:54:15.360844 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:15.360765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-cwvfk" event={"ID":"b06fdbbf-8b20-44bc-9924-b03cf338c82e","Type":"ContainerStarted","Data":"d372fedf29ab0864d0addb8142aea9472269fc787142d082ea1eacc4a5e47971"}
Jul 01 20:54:15.361190 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:15.360872    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:15.378634 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:15.378590    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-cwvfk" podStartSLOduration=1.622744161 podStartE2EDuration="4.37857588s" totalImagesPullingTime="2.755831719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:15.376808213 +0000 UTC m=+446.292086251" watchObservedRunningTime="2026-07-01 20:54:15.37857588 +0000 UTC m=+446.293853919"
Jul 01 20:54:26.368511 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:26.368486    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-cwvfk"
Jul 01 20:54:27.340004 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.339974    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bn8nq"]
Jul 01 20:54:27.343521 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.343506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.352384 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.352361    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bn8nq"]
Jul 01 20:54:27.423453 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.423429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5j9\" (UniqueName: \"kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.423749 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.423467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.524026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.524001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5j9\" (UniqueName: \"kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.524133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.524036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.524316 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.524303    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.533320 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.533300    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5j9\" (UniqueName: \"kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9\") pod \"s3-init-bn8nq\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") " pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.664607 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.664550    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:54:27.806097 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:27.806071    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bn8nq"]
Jul 01 20:54:27.809286 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:54:27.809241    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc523d6e4_de16_45d8_a73e_58df48bb156d.slice/crio-0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099 WatchSource:0}: Error finding container 0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099: Status 404 returned error can't find the container with id 0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099
Jul 01 20:54:28.408894 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:28.408857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerStarted","Data":"0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099"}
Jul 01 20:54:30.758752 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:30.758704    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:32.427655 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:32.427621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerStarted","Data":"bf6d6b9517f23112a6db86abea19e2b1ae1bbf859aee0455de6326acc4269e73"}
Jul 01 20:54:33.350344 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:33.350313    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:34.436403 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:34.436371    2574 generic.go:356] "Generic (PLEG): container finished" podID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerID="bf6d6b9517f23112a6db86abea19e2b1ae1bbf859aee0455de6326acc4269e73" exitCode=0
Jul 01 20:54:34.436787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:34.436422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerDied","Data":"bf6d6b9517f23112a6db86abea19e2b1ae1bbf859aee0455de6326acc4269e73"}
Jul 01 20:54:39.459362 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:39.459328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerStarted","Data":"0d96c9a6fbbc38d95c32f87a39ccc71e4ded7fcde020bd3b650705e9edf2e538"}
Jul 01 20:54:39.479749 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:39.479686    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bn8nq" podStartSLOduration=2.382133364 podStartE2EDuration="12.479670282s" totalImagesPullingTime="8.64744477s" totalInitContainerRuntime="1.450092148s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:39.478337078 +0000 UTC m=+470.393615115" watchObservedRunningTime="2026-07-01 20:54:39.479670282 +0000 UTC m=+470.394948319"
Jul 01 20:54:41.704686 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:41.704651    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:54:41.805728 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:54:41.805680    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:12.640126 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:12.640088    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:44.544019 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.543989    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/s3-init-bn8nq" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="s3-init" containerID="cri-o://0d96c9a6fbbc38d95c32f87a39ccc71e4ded7fcde020bd3b650705e9edf2e538" gracePeriod=30
Jul 01 20:55:44.693701 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.693674    2574 generic.go:356] "Generic (PLEG): container finished" podID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerID="0d96c9a6fbbc38d95c32f87a39ccc71e4ded7fcde020bd3b650705e9edf2e538" exitCode=0
Jul 01 20:55:44.693855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.693755    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerDied","Data":"0d96c9a6fbbc38d95c32f87a39ccc71e4ded7fcde020bd3b650705e9edf2e538"}
Jul 01 20:55:44.693855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.693807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bn8nq" event={"ID":"c523d6e4-de16-45d8-a73e-58df48bb156d","Type":"ContainerDied","Data":"0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099"}
Jul 01 20:55:44.693855 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.693823    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af04cd310f848169039cfa715aeb80222b9d0dbc1183272ee232f5ec6d95099"
Jul 01 20:55:44.694008 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:44.693893    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:55:45.697729 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.697690    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:55:45.704447 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.704423    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:55:45.830251 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.830226    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache\") pod \"c523d6e4-de16-45d8-a73e-58df48bb156d\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") "
Jul 01 20:55:45.830373 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.830284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf5j9\" (UniqueName: \"kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9\") pod \"c523d6e4-de16-45d8-a73e-58df48bb156d\" (UID: \"c523d6e4-de16-45d8-a73e-58df48bb156d\") "
Jul 01 20:55:45.832340 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.832311    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9" (OuterVolumeSpecName: "kube-api-access-vf5j9") pod "c523d6e4-de16-45d8-a73e-58df48bb156d" (UID: "c523d6e4-de16-45d8-a73e-58df48bb156d"). InnerVolumeSpecName "kube-api-access-vf5j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:55:45.848506 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.848483    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c523d6e4-de16-45d8-a73e-58df48bb156d" (UID: "c523d6e4-de16-45d8-a73e-58df48bb156d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:55:45.931079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.931054    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c523d6e4-de16-45d8-a73e-58df48bb156d-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:55:45.931079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:45.931077    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5j9\" (UniqueName: \"kubernetes.io/projected/c523d6e4-de16-45d8-a73e-58df48bb156d-kube-api-access-vf5j9\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:55:46.701957 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:46.701929    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bn8nq"
Jul 01 20:55:48.057256 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:48.057227    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:48.157165 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:48.157138    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:59.142391 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.142354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:59.267206 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267177    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:55:59.267456 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267446    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="s3-init"
Jul 01 20:55:59.267499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267458    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="s3-init"
Jul 01 20:55:59.267499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267467    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="download-hf-model"
Jul 01 20:55:59.267499 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267472    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="download-hf-model"
Jul 01 20:55:59.267583 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.267517    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c523d6e4-de16-45d8-a73e-58df48bb156d" containerName="s3-init"
Jul 01 20:55:59.269880 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.269863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.272168 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.272142    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 20:55:59.272304 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.272179    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-06549-predictor-serving-cert\""
Jul 01 20:55:59.272448 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.272433    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7bgz\""
Jul 01 20:55:59.272515 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.272469    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 20:55:59.272700 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.272653    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 20:55:59.273045 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.273030    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 20:55:59.273045 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.273045    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-06549-kube-rbac-proxy-sar-config\""
Jul 01 20:55:59.284765 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.284746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:55:59.326419 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.326393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:55:59.428636 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.428576    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw7j\" (UniqueName: \"kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.428636 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.428613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.428810 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.428662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.428810 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.428702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.428810 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.428753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw7j\" (UniqueName: \"kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529307 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529307 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529459 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529577 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529841 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529825    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.529899 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.529847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.531771 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.531752    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.545630 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.545602    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw7j\" (UniqueName: \"kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j\") pod \"isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.580083 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.580066    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:55:59.710171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.710106    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:55:59.712771 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:55:59.712749    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea36b617_d104_4fc4_919a_8c2eafa8edf6.slice/crio-f50b3aeb25afb20050c09fdf1cbc0bdc3074c20ecc018ea8c8bce06f167ffc1e WatchSource:0}: Error finding container f50b3aeb25afb20050c09fdf1cbc0bdc3074c20ecc018ea8c8bce06f167ffc1e: Status 404 returned error can't find the container with id f50b3aeb25afb20050c09fdf1cbc0bdc3074c20ecc018ea8c8bce06f167ffc1e
Jul 01 20:55:59.749661 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:55:59.749628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerStarted","Data":"f50b3aeb25afb20050c09fdf1cbc0bdc3074c20ecc018ea8c8bce06f167ffc1e"}
Jul 01 20:56:00.754482 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:00.754447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerStarted","Data":"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"}
Jul 01 20:56:03.767007 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:03.766975    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerID="75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818" exitCode=0
Jul 01 20:56:03.767007 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:03.767011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerDied","Data":"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"}
Jul 01 20:56:18.844091 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:18.844044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerStarted","Data":"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"}
Jul 01 20:56:20.852675 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:20.852639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerStarted","Data":"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"}
Jul 01 20:56:23.866371 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.866337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerStarted","Data":"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"}
Jul 01 20:56:23.866793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.866404    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:56:23.866793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.866431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:56:23.866793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.866460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:56:23.867878 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.867840    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:23.868483 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.868461    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:23.888395 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:23.888354    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podStartSLOduration=2.269139481 podStartE2EDuration="24.888342142s" totalImagesPullingTime="19.02080036s" totalInitContainerRuntime="3.598402301s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:55:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:56:23.886895206 +0000 UTC m=+574.802173270" watchObservedRunningTime="2026-07-01 20:56:23.888342142 +0000 UTC m=+574.803620180"
Jul 01 20:56:24.870170 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:24.870119    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:24.870606 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:24.870557    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:29.873929 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:29.873895    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:56:29.874518 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:29.874489    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:29.874754 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:29.874728    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:39.874692 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:39.874650    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:39.875122 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:39.875072    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:49.517749 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:49.517702    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:56:49.524780 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:49.524762    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 20:56:49.875283 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:49.875183    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:49.875690 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:49.875665    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:59.874967 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:59.874928    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:56:59.876489 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:56:59.875306    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:09.874482 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:09.874390    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:57:09.874997 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:09.874874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:19.874500 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:19.874456    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:57:19.875026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:19.874902    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:29.875640 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:29.875611    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:57:29.876112 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:29.875774    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:57:44.447684 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.447649    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:57:44.491589 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.491554    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:57:44.491875 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.491841    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" containerID="cri-o://291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf" gracePeriod=30
Jul 01 20:57:44.491875 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.491857    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" containerID="cri-o://715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467" gracePeriod=30
Jul 01 20:57:44.492089 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.491864    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" containerID="cri-o://440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec" gracePeriod=30
Jul 01 20:57:44.509444 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.509422    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:57:44.512143 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.512128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.514503 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.514483    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c682d-predictor-serving-cert\""
Jul 01 20:57:44.514620 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.514484    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c682d-kube-rbac-proxy-sar-config\""
Jul 01 20:57:44.525538 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.525517    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:57:44.574872 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.574842    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:57:44.577370 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.577354    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.579796 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.579769    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c682d-kube-rbac-proxy-sar-config\""
Jul 01 20:57:44.580160 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.580136    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c682d-predictor-serving-cert\""
Jul 01 20:57:44.589792 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.589769    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:57:44.697992 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.697918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.697992 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.697958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.698174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.698174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.698174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.698174 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7qb\" (UniqueName: \"kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.698299 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.698299 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5287w\" (UniqueName: \"kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.698299 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.698299 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.698287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.799504 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.799672 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.799672 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.799793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7qb\" (UniqueName: \"kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.799793 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.799896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5287w\" (UniqueName: \"kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.799896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.799896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.799896 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.800093 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.799930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.800334 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800312    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.800479 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800458    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.800479 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800340    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.800615 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800317    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.800615 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800385    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.800866 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.800845    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.802531 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.802511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.802597 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.802582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.810660 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.810631    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7qb\" (UniqueName: \"kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb\") pod \"isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.811617 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.811601    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5287w\" (UniqueName: \"kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w\") pod \"isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.821522 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.821492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:44.870536 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.870457    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:57:44.887637 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.887487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:57:44.959478 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:44.959452    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:57:44.962328 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:57:44.962301    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffa3d25_a6b4_4f0c_bded_76ca6fa5549b.slice/crio-59a314b6e3d8c0a53f58a6ff18e4fbc3b6427ffa12002c935321e906cd49b17a WatchSource:0}: Error finding container 59a314b6e3d8c0a53f58a6ff18e4fbc3b6427ffa12002c935321e906cd49b17a: Status 404 returned error can't find the container with id 59a314b6e3d8c0a53f58a6ff18e4fbc3b6427ffa12002c935321e906cd49b17a
Jul 01 20:57:45.025802 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.025779    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:57:45.029627 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:57:45.029607    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6534c9_552c_4170_a81e_d2dbcc14513c.slice/crio-71c22c867fa8096639f1afa1687c186a30721b6218a192667720308a97250178 WatchSource:0}: Error finding container 71c22c867fa8096639f1afa1687c186a30721b6218a192667720308a97250178: Status 404 returned error can't find the container with id 71c22c867fa8096639f1afa1687c186a30721b6218a192667720308a97250178
Jul 01 20:57:45.175805 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.175765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerStarted","Data":"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"}
Jul 01 20:57:45.176001 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.175844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerStarted","Data":"71c22c867fa8096639f1afa1687c186a30721b6218a192667720308a97250178"}
Jul 01 20:57:45.177606 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.177575    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerStarted","Data":"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"}
Jul 01 20:57:45.177771 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.177636    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerStarted","Data":"59a314b6e3d8c0a53f58a6ff18e4fbc3b6427ffa12002c935321e906cd49b17a"}
Jul 01 20:57:45.180556 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.180521    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerID="440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec" exitCode=2
Jul 01 20:57:45.180556 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:45.180554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerDied","Data":"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"}
Jul 01 20:57:46.470936 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:46.470907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:57:49.200639 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.200602    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerID="291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf" exitCode=0
Jul 01 20:57:49.201040 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.200680    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerDied","Data":"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"}
Jul 01 20:57:49.202259 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.202237    2574 generic.go:356] "Generic (PLEG): container finished" podID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerID="1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a" exitCode=0
Jul 01 20:57:49.202383 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.202314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerDied","Data":"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"}
Jul 01 20:57:49.204031 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.203995    2574 generic.go:356] "Generic (PLEG): container finished" podID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerID="341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd" exitCode=0
Jul 01 20:57:49.204128 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.204031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerDied","Data":"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"}
Jul 01 20:57:49.870362 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.870306    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:57:49.875145 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.875055    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:57:49.875400 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:49.875377    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:50.211151 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:50.211122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerStarted","Data":"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"}
Jul 01 20:57:50.211604 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:50.211192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerStarted","Data":"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"}
Jul 01 20:57:50.211604 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:50.211429    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:50.230927 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:50.230878    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podStartSLOduration=2.627956647 podStartE2EDuration="6.230861062s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.602904415s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:57:50.230220304 +0000 UTC m=+661.145498355" watchObservedRunningTime="2026-07-01 20:57:50.230861062 +0000 UTC m=+661.146139100"
Jul 01 20:57:51.217787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:51.217748    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:51.219171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:51.219139    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:57:52.224009 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:52.223968    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:57:54.870722 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:54.870673    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:57:54.871183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:54.870851    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:57:56.578270 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:56.578239    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:57:57.229281 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:57.228514    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:57:57.229281 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:57.229022    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:57:59.871185 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:59.871132    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:57:59.874443 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:59.874413    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:57:59.874901 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:57:59.874873    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:04.871188 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:04.871138    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:58:07.229380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:07.229342    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:08.296292 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:08.296258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerStarted","Data":"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"}
Jul 01 20:58:08.296676 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:08.296319    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerStarted","Data":"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"}
Jul 01 20:58:08.296676 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:08.296502    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:58:08.319477 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:08.319432    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podStartSLOduration=2.424811933 podStartE2EDuration="24.319417128s" totalImagesPullingTime="18.121317411s" totalInitContainerRuntime="3.773287784s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:58:08.317553424 +0000 UTC m=+679.232831472" watchObservedRunningTime="2026-07-01 20:58:08.319417128 +0000 UTC m=+679.234695165"
Jul 01 20:58:09.299864 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.299833    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:58:09.300976 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.300951    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:09.871256 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.871217    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:58:09.874539 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.874501    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 20:58:09.874676 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.874620    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:58:09.874846 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.874825    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:09.874927 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:09.874913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:58:10.303583 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:10.303543    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:14.870866 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:14.870780    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 20:58:15.167409 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.167387    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:58:15.270380 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270352    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls\") pod \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") "
Jul 01 20:58:15.270530 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270410    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config\") pod \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") "
Jul 01 20:58:15.270530 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert\") pod \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") "
Jul 01 20:58:15.270530 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location\") pod \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") "
Jul 01 20:58:15.270530 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhw7j\" (UniqueName: \"kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j\") pod \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\" (UID: \"ea36b617-d104-4fc4-919a-8c2eafa8edf6\") "
Jul 01 20:58:15.270861 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270835    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea36b617-d104-4fc4-919a-8c2eafa8edf6" (UID: "ea36b617-d104-4fc4-919a-8c2eafa8edf6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:58:15.270918 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270848    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea36b617-d104-4fc4-919a-8c2eafa8edf6" (UID: "ea36b617-d104-4fc4-919a-8c2eafa8edf6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:58:15.270918 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.270870    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea36b617-d104-4fc4-919a-8c2eafa8edf6" (UID: "ea36b617-d104-4fc4-919a-8c2eafa8edf6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:58:15.272688 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.272669    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j" (OuterVolumeSpecName: "kube-api-access-bhw7j") pod "ea36b617-d104-4fc4-919a-8c2eafa8edf6" (UID: "ea36b617-d104-4fc4-919a-8c2eafa8edf6"). InnerVolumeSpecName "kube-api-access-bhw7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:58:15.272782 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.272760    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea36b617-d104-4fc4-919a-8c2eafa8edf6" (UID: "ea36b617-d104-4fc4-919a-8c2eafa8edf6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:58:15.308159 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.308135    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:58:15.308606 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.308579    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:15.324663 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.324638    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerID="715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467" exitCode=0
Jul 01 20:58:15.324798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.324692    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"
Jul 01 20:58:15.324798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.324699    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerDied","Data":"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"}
Jul 01 20:58:15.324798 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.324785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh" event={"ID":"ea36b617-d104-4fc4-919a-8c2eafa8edf6","Type":"ContainerDied","Data":"f50b3aeb25afb20050c09fdf1cbc0bdc3074c20ecc018ea8c8bce06f167ffc1e"}
Jul 01 20:58:15.324962 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.324834    2574 scope.go:117] "RemoveContainer" containerID="715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"
Jul 01 20:58:15.333483 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.333470    2574 scope.go:117] "RemoveContainer" containerID="440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"
Jul 01 20:58:15.340762 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.340743    2574 scope.go:117] "RemoveContainer" containerID="291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"
Jul 01 20:58:15.347407 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.347387    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:58:15.347956 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.347944    2574 scope.go:117] "RemoveContainer" containerID="75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"
Jul 01 20:58:15.351845 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.351826    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh"]
Jul 01 20:58:15.354927 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.354910    2574 scope.go:117] "RemoveContainer" containerID="715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"
Jul 01 20:58:15.355210 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:58:15.355191    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467\": container with ID starting with 715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467 not found: ID does not exist" containerID="715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"
Jul 01 20:58:15.355258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355218    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467"} err="failed to get container status \"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467\": rpc error: code = NotFound desc = could not find container \"715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467\": container with ID starting with 715028d03ec093326d07aaaad43e99c0a6b4182df946c74bbab10d3b8806e467 not found: ID does not exist"
Jul 01 20:58:15.355258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355233    2574 scope.go:117] "RemoveContainer" containerID="440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"
Jul 01 20:58:15.355474 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:58:15.355457    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec\": container with ID starting with 440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec not found: ID does not exist" containerID="440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"
Jul 01 20:58:15.355543 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355482    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec"} err="failed to get container status \"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec\": rpc error: code = NotFound desc = could not find container \"440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec\": container with ID starting with 440188cee41c32468a023e56a12d0e692d9cd815276d4b60fb210f2d63fb49ec not found: ID does not exist"
Jul 01 20:58:15.355543 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355497    2574 scope.go:117] "RemoveContainer" containerID="291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"
Jul 01 20:58:15.355740 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:58:15.355704    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf\": container with ID starting with 291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf not found: ID does not exist" containerID="291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"
Jul 01 20:58:15.355787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355739    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf"} err="failed to get container status \"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf\": rpc error: code = NotFound desc = could not find container \"291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf\": container with ID starting with 291a02031b503c15295f33a3aa9d0461b1049866b6417f73e10b7dd728fc90cf not found: ID does not exist"
Jul 01 20:58:15.355787 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355752    2574 scope.go:117] "RemoveContainer" containerID="75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"
Jul 01 20:58:15.355994 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:58:15.355978    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818\": container with ID starting with 75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818 not found: ID does not exist" containerID="75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"
Jul 01 20:58:15.356033 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.355998    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818"} err="failed to get container status \"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818\": rpc error: code = NotFound desc = could not find container \"75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818\": container with ID starting with 75af1fa5d0f994a8b1bb49632fcd9499e543fc55d2a66cf424959a838d76a818 not found: ID does not exist"
Jul 01 20:58:15.371455 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.371434    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw7j\" (UniqueName: \"kubernetes.io/projected/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-api-access-bhw7j\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:58:15.371560 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.371459    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea36b617-d104-4fc4-919a-8c2eafa8edf6-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:58:15.371560 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.371474    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:58:15.371560 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.371487    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea36b617-d104-4fc4-919a-8c2eafa8edf6-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:58:15.371560 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.371500    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea36b617-d104-4fc4-919a-8c2eafa8edf6-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:58:15.547789 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:15.547761    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" path="/var/lib/kubelet/pods/ea36b617-d104-4fc4-919a-8c2eafa8edf6/volumes"
Jul 01 20:58:17.229322 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:17.229280    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:25.308946 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:25.308906    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:27.229485 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:27.229443    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:30.834673 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:30.834631    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:58:35.309028 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:35.308986    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:37.229479 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:37.229435    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:45.309101 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:45.309013    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:47.229068 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:47.229027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:55.309592 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:55.309545    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:58:57.229869 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:57.229831    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:58:57.448419 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:57.448385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:58:57.504350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:58:57.504283    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:05.309879 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:05.309847    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:59:07.230658 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:07.230630    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:59:24.469529 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:24.469500    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:24.907569 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:24.907484    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:34.542123 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.542088    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:34.707258 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.707222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:34.764426 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.764383    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:59:34.764698 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.764661    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" containerID="cri-o://2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be" gracePeriod=30
Jul 01 20:59:34.765009 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.764666    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kube-rbac-proxy" containerID="cri-o://1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b" gracePeriod=30
Jul 01 20:59:34.777918 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.777689    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 20:59:34.778026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778008    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent"
Jul 01 20:59:34.778026 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778019    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778028    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778034    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778048    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="storage-initializer"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778053    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="storage-initializer"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778060    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778065    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778105    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kserve-container"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778112    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="agent"
Jul 01 20:59:34.778133 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.778117    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea36b617-d104-4fc4-919a-8c2eafa8edf6" containerName="kube-rbac-proxy"
Jul 01 20:59:34.780427 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.780409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.782519 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.782495    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-1851d-predictor-serving-cert\""
Jul 01 20:59:34.782636 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.782619    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-1851d-kube-rbac-proxy-sar-config\""
Jul 01 20:59:34.793308 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.793260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 20:59:34.812832 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.812810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:34.849082 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.849051    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 20:59:34.851972 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.851957    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.854986 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.854970    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-1851d-kube-rbac-proxy-sar-config\""
Jul 01 20:59:34.855153 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.855137    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-1851d-predictor-serving-cert\""
Jul 01 20:59:34.865678 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.865657    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 20:59:34.878892 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.878868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.878989 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.878911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlx8\" (UniqueName: \"kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.878989 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.878942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.879071 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.879009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.879109 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.879077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.881836 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.881819    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:59:34.882035 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.882015    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" containerID="cri-o://4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831" gracePeriod=30
Jul 01 20:59:34.882130 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.882105    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kube-rbac-proxy" containerID="cri-o://ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb" gracePeriod=30
Jul 01 20:59:34.908410 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.908371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:34.979514 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tct6n\" (UniqueName: \"kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.979669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.979669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.979669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.979669 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.979912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.979912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:34.979912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlx8\" (UniqueName: \"kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.979912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.979912 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.979873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.980165 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.980008    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.980357 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.980330    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.980470 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.980454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.982484 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.982465    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:34.988620 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:34.988596    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlx8\" (UniqueName: \"kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8\") pod \"isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:35.007647 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.007615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:35.081275 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tct6n\" (UniqueName: \"kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.081275 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.081475 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.081475 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.081475 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.082272 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.081891    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.082272 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.082076    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.082272 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.082081    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.084606 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.084584    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.089955 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.089930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tct6n\" (UniqueName: \"kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n\") pod \"isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.090092 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.090077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:35.108825 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.108804    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 20:59:35.164025 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.163996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:35.229282 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.229212    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 20:59:35.234694 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:59:35.234670    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4715ca_b99e_4cb8_9606_d328c0253695.slice/crio-72900197ca5b92dac4c2ccaee97f22ff702f9f345facde9772da3fff33430c63 WatchSource:0}: Error finding container 72900197ca5b92dac4c2ccaee97f22ff702f9f345facde9772da3fff33430c63: Status 404 returned error can't find the container with id 72900197ca5b92dac4c2ccaee97f22ff702f9f345facde9772da3fff33430c63
Jul 01 20:59:35.302567 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.302542    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 20:59:35.304394 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.304360    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 01 20:59:35.305287 ip-10-0-135-188 kubenswrapper[2574]: W0701 20:59:35.305264    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc92d55d_3972_4ca7_8d50_2fa139ab3337.slice/crio-7e74d4756e27bf32f1c4946a4dc09ce505cc7d1c6513cb98394c10a564d15e3c WatchSource:0}: Error finding container 7e74d4756e27bf32f1c4946a4dc09ce505cc7d1c6513cb98394c10a564d15e3c: Status 404 returned error can't find the container with id 7e74d4756e27bf32f1c4946a4dc09ce505cc7d1c6513cb98394c10a564d15e3c
Jul 01 20:59:35.309061 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.309032    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 20:59:35.646893 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.646801    2574 generic.go:356] "Generic (PLEG): container finished" podID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerID="ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb" exitCode=2
Jul 01 20:59:35.646893 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.646833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerDied","Data":"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"}
Jul 01 20:59:35.649327 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.649303    2574 generic.go:356] "Generic (PLEG): container finished" podID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerID="1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b" exitCode=2
Jul 01 20:59:35.649471 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.649379    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerDied","Data":"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"}
Jul 01 20:59:35.651003 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.650981    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerStarted","Data":"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"}
Jul 01 20:59:35.651109 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.651030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerStarted","Data":"7e74d4756e27bf32f1c4946a4dc09ce505cc7d1c6513cb98394c10a564d15e3c"}
Jul 01 20:59:35.652534 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.652516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerStarted","Data":"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"}
Jul 01 20:59:35.652640 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:35.652554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerStarted","Data":"72900197ca5b92dac4c2ccaee97f22ff702f9f345facde9772da3fff33430c63"}
Jul 01 20:59:37.224173 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:37.224135    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 01 20:59:37.229507 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:37.229480    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 20:59:38.518853 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.518828    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:59:38.666457 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.666380    2574 generic.go:356] "Generic (PLEG): container finished" podID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerID="4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831" exitCode=0
Jul 01 20:59:38.666457 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.666417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerDied","Data":"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"}
Jul 01 20:59:38.666457 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.666432    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"
Jul 01 20:59:38.666653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.666475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm" event={"ID":"2e6534c9-552c-4170-a81e-d2dbcc14513c","Type":"ContainerDied","Data":"71c22c867fa8096639f1afa1687c186a30721b6218a192667720308a97250178"}
Jul 01 20:59:38.666653 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.666504    2574 scope.go:117] "RemoveContainer" containerID="ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"
Jul 01 20:59:38.674444 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.674428    2574 scope.go:117] "RemoveContainer" containerID="4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"
Jul 01 20:59:38.681589 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.681558    2574 scope.go:117] "RemoveContainer" containerID="1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"
Jul 01 20:59:38.688755 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.688696    2574 scope.go:117] "RemoveContainer" containerID="ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"
Jul 01 20:59:38.689005 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:38.688979    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb\": container with ID starting with ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb not found: ID does not exist" containerID="ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"
Jul 01 20:59:38.689079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.689005    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb"} err="failed to get container status \"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb\": rpc error: code = NotFound desc = could not find container \"ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb\": container with ID starting with ccb44543be768f60e946c332e0f4d04c166c7a5de36c7d7e8452089a19eeafcb not found: ID does not exist"
Jul 01 20:59:38.689079 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.689021    2574 scope.go:117] "RemoveContainer" containerID="4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"
Jul 01 20:59:38.689282 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:38.689261    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831\": container with ID starting with 4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831 not found: ID does not exist" containerID="4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"
Jul 01 20:59:38.689350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.689292    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831"} err="failed to get container status \"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831\": rpc error: code = NotFound desc = could not find container \"4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831\": container with ID starting with 4006061ce621e91b25d86185647638f467eaaa8dd1430959841614128f145831 not found: ID does not exist"
Jul 01 20:59:38.689350 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.689317    2574 scope.go:117] "RemoveContainer" containerID="1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"
Jul 01 20:59:38.689591 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:38.689574    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a\": container with ID starting with 1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a not found: ID does not exist" containerID="1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"
Jul 01 20:59:38.689649 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.689596    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a"} err="failed to get container status \"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a\": rpc error: code = NotFound desc = could not find container \"1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a\": container with ID starting with 1512aa654eec1bd9ccc3abb28971fb1e0be4324e5c67cfeb7579be1acd18d76a not found: ID does not exist"
Jul 01 20:59:38.711903 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.711878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert\") pod \"2e6534c9-552c-4170-a81e-d2dbcc14513c\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") "
Jul 01 20:59:38.712001 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.711936    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config\") pod \"2e6534c9-552c-4170-a81e-d2dbcc14513c\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") "
Jul 01 20:59:38.712001 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.711976    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location\") pod \"2e6534c9-552c-4170-a81e-d2dbcc14513c\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") "
Jul 01 20:59:38.712113 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712022    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7qb\" (UniqueName: \"kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb\") pod \"2e6534c9-552c-4170-a81e-d2dbcc14513c\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") "
Jul 01 20:59:38.712113 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712073    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls\") pod \"2e6534c9-552c-4170-a81e-d2dbcc14513c\" (UID: \"2e6534c9-552c-4170-a81e-d2dbcc14513c\") "
Jul 01 20:59:38.712279 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712255    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e6534c9-552c-4170-a81e-d2dbcc14513c" (UID: "2e6534c9-552c-4170-a81e-d2dbcc14513c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:59:38.712343 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712320    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e6534c9-552c-4170-a81e-d2dbcc14513c" (UID: "2e6534c9-552c-4170-a81e-d2dbcc14513c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:59:38.712343 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712322    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e6534c9-552c-4170-a81e-d2dbcc14513c" (UID: "2e6534c9-552c-4170-a81e-d2dbcc14513c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:59:38.712767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712630    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6534c9-552c-4170-a81e-d2dbcc14513c-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:38.712767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712655    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:38.712767 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.712670    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:38.714611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.714584    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e6534c9-552c-4170-a81e-d2dbcc14513c" (UID: "2e6534c9-552c-4170-a81e-d2dbcc14513c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:59:38.714857 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.714830    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb" (OuterVolumeSpecName: "kube-api-access-2z7qb") pod "2e6534c9-552c-4170-a81e-d2dbcc14513c" (UID: "2e6534c9-552c-4170-a81e-d2dbcc14513c"). InnerVolumeSpecName "kube-api-access-2z7qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:59:38.813468 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.813431    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6534c9-552c-4170-a81e-d2dbcc14513c-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:38.813468 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.813460    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7qb\" (UniqueName: \"kubernetes.io/projected/2e6534c9-552c-4170-a81e-d2dbcc14513c-kube-api-access-2z7qb\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:38.988325 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.988291    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:59:38.992245 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:38.992216    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm"]
Jul 01 20:59:39.401852 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.401831    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:59:39.519737 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.519688    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls\") pod \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") "
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.519769    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location\") pod \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") "
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.519790    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert\") pod \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") "
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.519826    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5287w\" (UniqueName: \"kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w\") pod \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") "
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.519865    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config\") pod \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\" (UID: \"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b\") "
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.520154    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" (UID: "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:59:39.520183 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.520179    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" (UID: "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:59:39.520465 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.520249    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" (UID: "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:59:39.522145 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.522119    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" (UID: "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:59:39.522145 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.522119    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w" (OuterVolumeSpecName: "kube-api-access-5287w") pod "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" (UID: "fffa3d25-a6b4-4f0c-bded-76ca6fa5549b"). InnerVolumeSpecName "kube-api-access-5287w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:59:39.548447 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.548419    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" path="/var/lib/kubelet/pods/2e6534c9-552c-4170-a81e-d2dbcc14513c/volumes"
Jul 01 20:59:39.621152 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.621122    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:39.621152 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.621149    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:39.621358 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.621162    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:39.621358 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.621172    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:39.621358 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.621180    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5287w\" (UniqueName: \"kubernetes.io/projected/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b-kube-api-access-5287w\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 20:59:39.673572 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.673546    2574 generic.go:356] "Generic (PLEG): container finished" podID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerID="2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be" exitCode=0
Jul 01 20:59:39.673775 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.673599    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"
Jul 01 20:59:39.673775 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.673612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerDied","Data":"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"}
Jul 01 20:59:39.673775 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.673669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c" event={"ID":"fffa3d25-a6b4-4f0c-bded-76ca6fa5549b","Type":"ContainerDied","Data":"59a314b6e3d8c0a53f58a6ff18e4fbc3b6427ffa12002c935321e906cd49b17a"}
Jul 01 20:59:39.673775 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.673706    2574 scope.go:117] "RemoveContainer" containerID="1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"
Jul 01 20:59:39.675371 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.675346    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerID="6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e" exitCode=0
Jul 01 20:59:39.675476 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.675417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerDied","Data":"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"}
Jul 01 20:59:39.677082 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.677062    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerID="9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683" exitCode=0
Jul 01 20:59:39.677171 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.677091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerDied","Data":"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"}
Jul 01 20:59:39.684630 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.684617    2574 scope.go:117] "RemoveContainer" containerID="2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"
Jul 01 20:59:39.694589 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.694557    2574 scope.go:117] "RemoveContainer" containerID="341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"
Jul 01 20:59:39.718044 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.718017    2574 scope.go:117] "RemoveContainer" containerID="1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"
Jul 01 20:59:39.718462 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:39.718433    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b\": container with ID starting with 1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b not found: ID does not exist" containerID="1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"
Jul 01 20:59:39.718566 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.718463    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b"} err="failed to get container status \"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b\": rpc error: code = NotFound desc = could not find container \"1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b\": container with ID starting with 1869daeb05b0c7db95cdf727f0217ffcceab8960de604a78cad1b5bc4e1aaf0b not found: ID does not exist"
Jul 01 20:59:39.718566 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.718484    2574 scope.go:117] "RemoveContainer" containerID="2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"
Jul 01 20:59:39.718815 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:39.718784    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be\": container with ID starting with 2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be not found: ID does not exist" containerID="2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"
Jul 01 20:59:39.718904 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.718809    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be"} err="failed to get container status \"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be\": rpc error: code = NotFound desc = could not find container \"2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be\": container with ID starting with 2580b66cddc09e3fb0ee964d2fdb25cbf52772261c7cfe2b4f73f6bfa8d502be not found: ID does not exist"
Jul 01 20:59:39.718904 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.718831    2574 scope.go:117] "RemoveContainer" containerID="341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"
Jul 01 20:59:39.719146 ip-10-0-135-188 kubenswrapper[2574]: E0701 20:59:39.719123    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd\": container with ID starting with 341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd not found: ID does not exist" containerID="341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"
Jul 01 20:59:39.719217 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.719155    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd"} err="failed to get container status \"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd\": rpc error: code = NotFound desc = could not find container \"341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd\": container with ID starting with 341cc5bf6c212c3b4921ee802c3a7b564fa5fd6360a0885ddd4d641cc3312dbd not found: ID does not exist"
Jul 01 20:59:39.728929 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.728906    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:59:39.734758 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:39.734706    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c"]
Jul 01 20:59:40.685482 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.685444    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerStarted","Data":"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"}
Jul 01 20:59:40.686087 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.685545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerStarted","Data":"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"}
Jul 01 20:59:40.686087 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.685911    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:40.686087 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.686070    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:40.687443 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.687417    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 20:59:40.688076 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.688052    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerStarted","Data":"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"}
Jul 01 20:59:40.688162 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.688113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerStarted","Data":"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"}
Jul 01 20:59:40.688351 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.688331    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:40.705194 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.705142    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podStartSLOduration=3.001060524 podStartE2EDuration="6.705127889s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704067365s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:59:40.703541449 +0000 UTC m=+771.618819523" watchObservedRunningTime="2026-07-01 20:59:40.705127889 +0000 UTC m=+771.620405926"
Jul 01 20:59:40.722938 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:40.722890    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podStartSLOduration=3.182187428 podStartE2EDuration="6.722874195s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.540686767s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:59:40.721201642 +0000 UTC m=+771.636479679" watchObservedRunningTime="2026-07-01 20:59:40.722874195 +0000 UTC m=+771.638152232"
Jul 01 20:59:41.553282 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:41.553244    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" path="/var/lib/kubelet/pods/fffa3d25-a6b4-4f0c-bded-76ca6fa5549b/volumes"
Jul 01 20:59:41.692513 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:41.692470    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 20:59:41.692972 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:41.692526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:41.693769 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:41.693746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 20:59:42.696638 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:42.696594    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 20:59:46.696952 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:46.696922    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 20:59:46.697462 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:46.697435    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 20:59:47.707795 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:47.707769    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 20:59:47.708466 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:47.708439    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 20:59:56.697611 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:56.697569    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 20:59:57.708507 ip-10-0-135-188 kubenswrapper[2574]: I0701 20:59:57.708471    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:06.698054 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:06.698014    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 21:00:07.708593 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:07.708552    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:16.698010 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:16.697974    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 21:00:17.708840 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:17.708804    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:26.697494 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:26.697458    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 21:00:27.709126 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:27.709090    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:36.698658 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:36.698628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 21:00:37.708700 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:37.708660    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:47.709128 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:47.709088    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:00:57.709110 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:00:57.709082    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 21:01:14.846018 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:14.845980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:15.032795 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:15.032764    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:24.821567 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:24.821534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:24.965819 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:24.965782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:25.002799 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.002771    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:01:25.003077 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003066    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003079    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003090    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003096    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003106    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003111    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003119    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003124    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003133    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="storage-initializer"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003139    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="storage-initializer"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003145    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="storage-initializer"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003150    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="storage-initializer"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003195    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003204    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kube-rbac-proxy"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003211    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffa3d25-a6b4-4f0c-bded-76ca6fa5549b" containerName="kserve-container"
Jul 01 21:01:25.003345 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.003219    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6534c9-552c-4170-a81e-d2dbcc14513c" containerName="kserve-container"
Jul 01 21:01:25.005239 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.005224    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.007603 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.007581    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1255c-predictor-serving-cert\""
Jul 01 21:01:25.007792 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.007646    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1255c-kube-rbac-proxy-sar-config\""
Jul 01 21:01:25.015261 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.015242    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:01:25.049233 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.049207    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 21:01:25.049476 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.049439    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" containerID="cri-o://893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079" gracePeriod=30
Jul 01 21:01:25.049567 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.049484    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kube-rbac-proxy" containerID="cri-o://d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67" gracePeriod=30
Jul 01 21:01:25.075655 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.075582    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:25.090808 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.090780    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 21:01:25.091076 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.091051    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" containerID="cri-o://8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab" gracePeriod=30
Jul 01 21:01:25.091132 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.091088    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kube-rbac-proxy" containerID="cri-o://24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693" gracePeriod=30
Jul 01 21:01:25.165142 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.165110    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.165258 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.165169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.165258 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.165219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrxv\" (UniqueName: \"kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.172594 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.172572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:25.266562 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.266529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.266676 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.266569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.266676 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.266601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrxv\" (UniqueName: \"kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.266811 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:25.266672    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-serving-cert: secret "message-dumper-raw-1255c-predictor-serving-cert" not found
Jul 01 21:01:25.266811 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:25.266759    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls podName:97566295-827d-4bd2-8a44-231e50c2bce0 nodeName:}" failed. No retries permitted until 2026-07-01 21:01:25.766739636 +0000 UTC m=+876.682017653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls") pod "message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" (UID: "97566295-827d-4bd2-8a44-231e50c2bce0") : secret "message-dumper-raw-1255c-predictor-serving-cert" not found
Jul 01 21:01:25.267239 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.267221    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.275734 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.275685    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrxv\" (UniqueName: \"kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.330421 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.330347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:25.771318 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.771287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.773653 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.773623    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") pod \"message-dumper-raw-1255c-predictor-cc944bc4-sxm5p\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") " pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:25.915018 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:25.914983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:26.045391 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.045368    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:01:26.047899 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:01:26.047878    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97566295_827d_4bd2_8a44_231e50c2bce0.slice/crio-17ae19e3b91409d0cb65bdf7d6a50d50de4fe112867c367603583391e8cf709b WatchSource:0}: Error finding container 17ae19e3b91409d0cb65bdf7d6a50d50de4fe112867c367603583391e8cf709b: Status 404 returned error can't find the container with id 17ae19e3b91409d0cb65bdf7d6a50d50de4fe112867c367603583391e8cf709b
Jul 01 21:01:26.049706 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.049693    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 21:01:26.101389 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.101360    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerID="24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693" exitCode=2
Jul 01 21:01:26.101590 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.101435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerDied","Data":"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"}
Jul 01 21:01:26.109041 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.109014    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerID="d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67" exitCode=2
Jul 01 21:01:26.109169 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.109085    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerDied","Data":"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"}
Jul 01 21:01:26.110244 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.110221    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerStarted","Data":"17ae19e3b91409d0cb65bdf7d6a50d50de4fe112867c367603583391e8cf709b"}
Jul 01 21:01:26.692619 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.692577    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 21:01:26.697947 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:26.697914    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 21:01:27.697071 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:27.697040    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 21:01:27.708857 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:27.708826    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 21:01:28.120696 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.120613    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerStarted","Data":"d960abe95a59ab4a0b82500400656ca3d9737d1727af69e405df74a01292e27b"}
Jul 01 21:01:28.120696 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.120674    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerStarted","Data":"5454da6ec8dc118896125c165db7c044103878626fce05e2ea62b52487dd7b72"}
Jul 01 21:01:28.120889 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.120703    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:28.120889 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.120770    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:28.122452 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.122434    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:28.140692 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.140640    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" podStartSLOduration=2.964843875 podStartE2EDuration="4.140622945s" totalImagesPullingTime="1.17577907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:01:28.139873419 +0000 UTC m=+879.055151458" watchObservedRunningTime="2026-07-01 21:01:28.140622945 +0000 UTC m=+879.055900995"
Jul 01 21:01:28.734752 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.734729    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 21:01:28.899964 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.899872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls\") pod \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") "
Jul 01 21:01:28.899964 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.899927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config\") pod \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") "
Jul 01 21:01:28.899964 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.899949    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert\") pod \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") "
Jul 01 21:01:28.900266 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.899987    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location\") pod \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") "
Jul 01 21:01:28.900266 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.900018    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tct6n\" (UniqueName: \"kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n\") pod \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\" (UID: \"dc92d55d-3972-4ca7-8d50-2fa139ab3337\") "
Jul 01 21:01:28.900371 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.900310    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc92d55d-3972-4ca7-8d50-2fa139ab3337" (UID: "dc92d55d-3972-4ca7-8d50-2fa139ab3337"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:01:28.900371 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.900337    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc92d55d-3972-4ca7-8d50-2fa139ab3337" (UID: "dc92d55d-3972-4ca7-8d50-2fa139ab3337"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:01:28.900371 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.900343    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc92d55d-3972-4ca7-8d50-2fa139ab3337" (UID: "dc92d55d-3972-4ca7-8d50-2fa139ab3337"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:01:28.902196 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.902174    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc92d55d-3972-4ca7-8d50-2fa139ab3337" (UID: "dc92d55d-3972-4ca7-8d50-2fa139ab3337"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:01:28.902276 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:28.902214    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n" (OuterVolumeSpecName: "kube-api-access-tct6n") pod "dc92d55d-3972-4ca7-8d50-2fa139ab3337" (UID: "dc92d55d-3972-4ca7-8d50-2fa139ab3337"). InnerVolumeSpecName "kube-api-access-tct6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:01:29.001158 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.001125    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.001158 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.001150    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.001158 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.001161    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tct6n\" (UniqueName: \"kubernetes.io/projected/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-api-access-tct6n\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.001359 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.001171    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc92d55d-3972-4ca7-8d50-2fa139ab3337-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.001359 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.001180    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc92d55d-3972-4ca7-8d50-2fa139ab3337-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.126621 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.126584    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerID="8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab" exitCode=0
Jul 01 21:01:29.126826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.126643    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"
Jul 01 21:01:29.126826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.126668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerDied","Data":"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"}
Jul 01 21:01:29.126826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.126748    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn" event={"ID":"dc92d55d-3972-4ca7-8d50-2fa139ab3337","Type":"ContainerDied","Data":"7e74d4756e27bf32f1c4946a4dc09ce505cc7d1c6513cb98394c10a564d15e3c"}
Jul 01 21:01:29.126826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.126777    2574 scope.go:117] "RemoveContainer" containerID="24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"
Jul 01 21:01:29.135576 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.135559    2574 scope.go:117] "RemoveContainer" containerID="8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"
Jul 01 21:01:29.142939 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.142916    2574 scope.go:117] "RemoveContainer" containerID="6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"
Jul 01 21:01:29.149365 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.149341    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 21:01:29.150423 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.150407    2574 scope.go:117] "RemoveContainer" containerID="24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"
Jul 01 21:01:29.150678 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:29.150661    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693\": container with ID starting with 24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693 not found: ID does not exist" containerID="24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"
Jul 01 21:01:29.150739 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.150686    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693"} err="failed to get container status \"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693\": rpc error: code = NotFound desc = could not find container \"24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693\": container with ID starting with 24856e0b6f8be731a7c0324b1967c2fa624bbad09657800eb242748eb211f693 not found: ID does not exist"
Jul 01 21:01:29.150739 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.150703    2574 scope.go:117] "RemoveContainer" containerID="8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"
Jul 01 21:01:29.150972 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:29.150954    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab\": container with ID starting with 8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab not found: ID does not exist" containerID="8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"
Jul 01 21:01:29.151033 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.150980    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab"} err="failed to get container status \"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab\": rpc error: code = NotFound desc = could not find container \"8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab\": container with ID starting with 8e3077a122378f8e88efca1a9284efa5d6fd314aaca58ef63b9d34773e3774ab not found: ID does not exist"
Jul 01 21:01:29.151033 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.151005    2574 scope.go:117] "RemoveContainer" containerID="6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"
Jul 01 21:01:29.151275 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:29.151259    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e\": container with ID starting with 6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e not found: ID does not exist" containerID="6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"
Jul 01 21:01:29.151315 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.151280    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e"} err="failed to get container status \"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e\": rpc error: code = NotFound desc = could not find container \"6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e\": container with ID starting with 6f56af16d090252507320b8977b6eab9b3ce98e2466e8397e7b8ac9f6655643e not found: ID does not exist"
Jul 01 21:01:29.153807 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.153790    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn"]
Jul 01 21:01:29.549614 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.549587    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" path="/var/lib/kubelet/pods/dc92d55d-3972-4ca7-8d50-2fa139ab3337/volumes"
Jul 01 21:01:29.594079 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.594051    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 21:01:29.706133 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdlx8\" (UniqueName: \"kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8\") pod \"dd4715ca-b99e-4cb8-9606-d328c0253695\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") "
Jul 01 21:01:29.706313 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706152    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config\") pod \"dd4715ca-b99e-4cb8-9606-d328c0253695\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") "
Jul 01 21:01:29.706313 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706205    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls\") pod \"dd4715ca-b99e-4cb8-9606-d328c0253695\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") "
Jul 01 21:01:29.706313 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location\") pod \"dd4715ca-b99e-4cb8-9606-d328c0253695\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") "
Jul 01 21:01:29.706313 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706290    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert\") pod \"dd4715ca-b99e-4cb8-9606-d328c0253695\" (UID: \"dd4715ca-b99e-4cb8-9606-d328c0253695\") "
Jul 01 21:01:29.706616 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706594    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd4715ca-b99e-4cb8-9606-d328c0253695" (UID: "dd4715ca-b99e-4cb8-9606-d328c0253695"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:01:29.706683 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706595    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd4715ca-b99e-4cb8-9606-d328c0253695" (UID: "dd4715ca-b99e-4cb8-9606-d328c0253695"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:01:29.706832 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.706810    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd4715ca-b99e-4cb8-9606-d328c0253695" (UID: "dd4715ca-b99e-4cb8-9606-d328c0253695"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:01:29.708383 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.708364    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd4715ca-b99e-4cb8-9606-d328c0253695" (UID: "dd4715ca-b99e-4cb8-9606-d328c0253695"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:01:29.708383 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.708370    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8" (OuterVolumeSpecName: "kube-api-access-jdlx8") pod "dd4715ca-b99e-4cb8-9606-d328c0253695" (UID: "dd4715ca-b99e-4cb8-9606-d328c0253695"). InnerVolumeSpecName "kube-api-access-jdlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:01:29.807650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.807565    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4715ca-b99e-4cb8-9606-d328c0253695-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.807650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.807598    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd4715ca-b99e-4cb8-9606-d328c0253695-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.807650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.807610    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.807650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.807619    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlx8\" (UniqueName: \"kubernetes.io/projected/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-api-access-jdlx8\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:29.807650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:29.807626    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4715ca-b99e-4cb8-9606-d328c0253695-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:01:30.132623 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.132540    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerID="893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079" exitCode=0
Jul 01 21:01:30.132623 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.132590    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"
Jul 01 21:01:30.132856 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.132625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerDied","Data":"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"}
Jul 01 21:01:30.132856 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.132683    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r" event={"ID":"dd4715ca-b99e-4cb8-9606-d328c0253695","Type":"ContainerDied","Data":"72900197ca5b92dac4c2ccaee97f22ff702f9f345facde9772da3fff33430c63"}
Jul 01 21:01:30.132856 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.132727    2574 scope.go:117] "RemoveContainer" containerID="d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"
Jul 01 21:01:30.142855 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.142840    2574 scope.go:117] "RemoveContainer" containerID="893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"
Jul 01 21:01:30.150059 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.150032    2574 scope.go:117] "RemoveContainer" containerID="9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"
Jul 01 21:01:30.155822 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.155800    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 21:01:30.157566 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.157547    2574 scope.go:117] "RemoveContainer" containerID="d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"
Jul 01 21:01:30.157943 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:30.157918    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67\": container with ID starting with d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67 not found: ID does not exist" containerID="d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"
Jul 01 21:01:30.158040 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.157951    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67"} err="failed to get container status \"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67\": rpc error: code = NotFound desc = could not find container \"d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67\": container with ID starting with d62c1a82d98e06e827bafc4b03946d124a9efe8a7f31c88a0aeac1ad5889bf67 not found: ID does not exist"
Jul 01 21:01:30.158040 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.157975    2574 scope.go:117] "RemoveContainer" containerID="893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"
Jul 01 21:01:30.158458 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:30.158434    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079\": container with ID starting with 893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079 not found: ID does not exist" containerID="893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"
Jul 01 21:01:30.158537 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.158462    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079"} err="failed to get container status \"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079\": rpc error: code = NotFound desc = could not find container \"893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079\": container with ID starting with 893c865e8f281f6f122977b48eec4ca027eb9cb45b82f8f0f66c240d8484e079 not found: ID does not exist"
Jul 01 21:01:30.158537 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.158478    2574 scope.go:117] "RemoveContainer" containerID="9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"
Jul 01 21:01:30.158757 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:01:30.158735    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683\": container with ID starting with 9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683 not found: ID does not exist" containerID="9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"
Jul 01 21:01:30.158807 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.158765    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683"} err="failed to get container status \"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683\": rpc error: code = NotFound desc = could not find container \"9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683\": container with ID starting with 9be40d58bad757422129540ad93410baf458907de60d0d24cb3bff09ee36a683 not found: ID does not exist"
Jul 01 21:01:30.159754 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:30.159737    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r"]
Jul 01 21:01:31.547818 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:31.547775    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" path="/var/lib/kubelet/pods/dd4715ca-b99e-4cb8-9606-d328c0253695/volumes"
Jul 01 21:01:35.137896 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:35.137862    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:01:44.992508 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:44.992437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:45.033874 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.033843    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:01:45.034302 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034282    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container"
Jul 01 21:01:45.034351 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034308    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container"
Jul 01 21:01:45.034351 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034319    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container"
Jul 01 21:01:45.034351 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034328    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034357    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034367    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034379    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="storage-initializer"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034388    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="storage-initializer"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034405    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034414    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034425    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="storage-initializer"
Jul 01 21:01:45.034449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034432    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="storage-initializer"
Jul 01 21:01:45.034965 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034499    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034965 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034513    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92d55d-3972-4ca7-8d50-2fa139ab3337" containerName="kserve-container"
Jul 01 21:01:45.034965 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034520    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kube-rbac-proxy"
Jul 01 21:01:45.034965 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.034526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4715ca-b99e-4cb8-9606-d328c0253695" containerName="kserve-container"
Jul 01 21:01:45.036928 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.036910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.039248 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.039227    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1255c-kube-rbac-proxy-sar-config\""
Jul 01 21:01:45.039462 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.039446    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1255c-predictor-serving-cert\""
Jul 01 21:01:45.039546 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.039485    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:01:45.039546 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.039526    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:01:45.049137 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.049114    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:01:45.073046 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.072955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:01:45.126773 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.126748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.126914 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.126789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8ch\" (UniqueName: \"kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.126914 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.126807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.126984 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.126904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.126984 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.126946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228056 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228253 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8ch\" (UniqueName: \"kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228253 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228253 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228253 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228573 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228556    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228868 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228846    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.228960 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.228849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.230767 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.230750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.237078 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.237054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8ch\" (UniqueName: \"kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch\") pod \"isvc-logger-raw-1255c-predictor-65576c9ff7-gl997\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.347280 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.347192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:45.475626 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:45.475602    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:01:45.478047 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:01:45.478015    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b4c1eb_942a_4812_8d7d_96dd48e6635e.slice/crio-5c264e7cac74a8da90ec03714a6e03f35cbd7a28be45c4885f185f75997d8561 WatchSource:0}: Error finding container 5c264e7cac74a8da90ec03714a6e03f35cbd7a28be45c4885f185f75997d8561: Status 404 returned error can't find the container with id 5c264e7cac74a8da90ec03714a6e03f35cbd7a28be45c4885f185f75997d8561
Jul 01 21:01:46.200535 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:46.200501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerStarted","Data":"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"}
Jul 01 21:01:46.200944 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:46.200562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerStarted","Data":"5c264e7cac74a8da90ec03714a6e03f35cbd7a28be45c4885f185f75997d8561"}
Jul 01 21:01:49.213944 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:49.213912    2574 generic.go:356] "Generic (PLEG): container finished" podID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerID="e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c" exitCode=0
Jul 01 21:01:49.214427 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:49.213989    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerDied","Data":"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"}
Jul 01 21:01:49.549236 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:49.549159    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:01:49.553826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:49.553805    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:01:50.219924 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.219892    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerStarted","Data":"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"}
Jul 01 21:01:50.220324 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.219955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerStarted","Data":"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"}
Jul 01 21:01:50.220324 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.219979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerStarted","Data":"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"}
Jul 01 21:01:50.220324 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.220275    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:50.220485 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.220422    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:50.221557 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.221531    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:01:50.242100 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:50.242057    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podStartSLOduration=1.843632613 podStartE2EDuration="5.242043836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.398411223s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:01:50.240494257 +0000 UTC m=+901.155772308" watchObservedRunningTime="2026-07-01 21:01:50.242043836 +0000 UTC m=+901.157321873"
Jul 01 21:01:51.224438 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:51.224403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:51.224886 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:51.224569    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:01:51.225621 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:51.225595    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:01:52.228096 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:52.228054    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:01:52.228632 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:52.228608    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:01:57.232471 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:57.232443    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:01:57.232985 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:57.232907    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:01:57.233267 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:01:57.233232    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:07.233143 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:07.233101    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:07.234668 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:07.233449    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:17.233266 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:17.233228    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:17.233785 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:17.233745    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:27.233426 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:27.233373    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:27.233918 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:27.233895    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:35.430933 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:35.430897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:02:37.233671 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:37.233626    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:37.234157 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:37.234138    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:47.233860 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:47.233822    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:47.234351 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:47.234324    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:02:57.233163 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:57.233126    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:02:57.233661 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:02:57.233637    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:03:06.544446 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:06.544413    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:06.544940 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:06.544925    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:20.022072 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.022040    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-1255c-predictor-cc944bc4-sxm5p_97566295-827d-4bd2-8a44-231e50c2bce0/kserve-container/0.log"
Jul 01 21:03:20.286430 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.286356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:03:20.350441 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.350408    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:03:20.350734 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.350690    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" containerID="cri-o://d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703" gracePeriod=30
Jul 01 21:03:20.350814 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.350742    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" containerID="cri-o://898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df" gracePeriod=30
Jul 01 21:03:20.350869 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.350750    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" containerID="cri-o://c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663" gracePeriod=30
Jul 01 21:03:20.365798 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.365775    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:03:20.368438 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.368419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.372302 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.372279    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a940b-predictor-serving-cert\""
Jul 01 21:03:20.372611 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.372590    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a940b-kube-rbac-proxy-sar-config\""
Jul 01 21:03:20.384887 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.384865    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:03:20.406162 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.406141    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:03:20.406483 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.406456    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kserve-container" containerID="cri-o://5454da6ec8dc118896125c165db7c044103878626fce05e2ea62b52487dd7b72" gracePeriod=30
Jul 01 21:03:20.406633 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.406496    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kube-rbac-proxy" containerID="cri-o://d960abe95a59ab4a0b82500400656ca3d9737d1727af69e405df74a01292e27b" gracePeriod=30
Jul 01 21:03:20.490291 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.490270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzkl\" (UniqueName: \"kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.490407 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.490326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.490407 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.490374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.490518 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.490408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.490600 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.490522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.580282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.580252    2574 generic.go:356] "Generic (PLEG): container finished" podID="97566295-827d-4bd2-8a44-231e50c2bce0" containerID="d960abe95a59ab4a0b82500400656ca3d9737d1727af69e405df74a01292e27b" exitCode=2
Jul 01 21:03:20.580282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.580279    2574 generic.go:356] "Generic (PLEG): container finished" podID="97566295-827d-4bd2-8a44-231e50c2bce0" containerID="5454da6ec8dc118896125c165db7c044103878626fce05e2ea62b52487dd7b72" exitCode=2
Jul 01 21:03:20.580471 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.580321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerDied","Data":"d960abe95a59ab4a0b82500400656ca3d9737d1727af69e405df74a01292e27b"}
Jul 01 21:03:20.580471 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.580389    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerDied","Data":"5454da6ec8dc118896125c165db7c044103878626fce05e2ea62b52487dd7b72"}
Jul 01 21:03:20.583224 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.583199    2574 generic.go:356] "Generic (PLEG): container finished" podID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerID="c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663" exitCode=2
Jul 01 21:03:20.583351 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.583274    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerDied","Data":"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"}
Jul 01 21:03:20.591112 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.591089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzkl\" (UniqueName: \"kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.591228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.591167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.591228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.591199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.591339 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.591242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.591339 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.591309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.592094 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.592066    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.592214 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.592076    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.592434 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.592407    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.594358 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.594331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.601216 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.601179    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzkl\" (UniqueName: \"kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl\") pod \"isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.639114 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.639094    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:03:20.679324 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.679301    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:20.792704 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.792670    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") pod \"97566295-827d-4bd2-8a44-231e50c2bce0\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") "
Jul 01 21:03:20.792874 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.792771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmrxv\" (UniqueName: \"kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv\") pod \"97566295-827d-4bd2-8a44-231e50c2bce0\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") "
Jul 01 21:03:20.792929 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.792881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config\") pod \"97566295-827d-4bd2-8a44-231e50c2bce0\" (UID: \"97566295-827d-4bd2-8a44-231e50c2bce0\") "
Jul 01 21:03:20.793444 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.793416    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97566295-827d-4bd2-8a44-231e50c2bce0" (UID: "97566295-827d-4bd2-8a44-231e50c2bce0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:03:20.795370 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.795342    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97566295-827d-4bd2-8a44-231e50c2bce0" (UID: "97566295-827d-4bd2-8a44-231e50c2bce0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:03:20.795459 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.795374    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv" (OuterVolumeSpecName: "kube-api-access-mmrxv") pod "97566295-827d-4bd2-8a44-231e50c2bce0" (UID: "97566295-827d-4bd2-8a44-231e50c2bce0"). InnerVolumeSpecName "kube-api-access-mmrxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:03:20.828149 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.828124    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:03:20.830187 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:03:20.830163    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7dc66b_dfc8_48d3_8e10_024fc08f7cd6.slice/crio-ddf23ae0bce8b12f8ce4906d2f676f06c57a0dc761680bee97774b9dd41b5b17 WatchSource:0}: Error finding container ddf23ae0bce8b12f8ce4906d2f676f06c57a0dc761680bee97774b9dd41b5b17: Status 404 returned error can't find the container with id ddf23ae0bce8b12f8ce4906d2f676f06c57a0dc761680bee97774b9dd41b5b17
Jul 01 21:03:20.893452 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.893429    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrxv\" (UniqueName: \"kubernetes.io/projected/97566295-827d-4bd2-8a44-231e50c2bce0-kube-api-access-mmrxv\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:20.893548 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.893454    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97566295-827d-4bd2-8a44-231e50c2bce0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:20.893548 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:20.893468    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97566295-827d-4bd2-8a44-231e50c2bce0-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:21.590637 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.590602    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerStarted","Data":"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"}
Jul 01 21:03:21.591082 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.590678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerStarted","Data":"ddf23ae0bce8b12f8ce4906d2f676f06c57a0dc761680bee97774b9dd41b5b17"}
Jul 01 21:03:21.592346 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.592321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p" event={"ID":"97566295-827d-4bd2-8a44-231e50c2bce0","Type":"ContainerDied","Data":"17ae19e3b91409d0cb65bdf7d6a50d50de4fe112867c367603583391e8cf709b"}
Jul 01 21:03:21.592451 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.592381    2574 scope.go:117] "RemoveContainer" containerID="d960abe95a59ab4a0b82500400656ca3d9737d1727af69e405df74a01292e27b"
Jul 01 21:03:21.592451 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.592328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"
Jul 01 21:03:21.600494 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.600476    2574 scope.go:117] "RemoveContainer" containerID="5454da6ec8dc118896125c165db7c044103878626fce05e2ea62b52487dd7b72"
Jul 01 21:03:21.624251 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.624195    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:03:21.628238 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:21.628218    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p"]
Jul 01 21:03:22.123195 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:22.123164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:03:22.229284 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:22.229230    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:23.548585 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:23.548543    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" path="/var/lib/kubelet/pods/97566295-827d-4bd2-8a44-231e50c2bce0/volumes"
Jul 01 21:03:24.606348 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:24.606322    2574 generic.go:356] "Generic (PLEG): container finished" podID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerID="2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db" exitCode=0
Jul 01 21:03:24.606757 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:24.606400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerDied","Data":"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"}
Jul 01 21:03:24.608986 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:24.608964    2574 generic.go:356] "Generic (PLEG): container finished" podID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerID="d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703" exitCode=0
Jul 01 21:03:24.609074 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:24.608994    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerDied","Data":"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"}
Jul 01 21:03:25.614695 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.614663    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerStarted","Data":"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"}
Jul 01 21:03:25.615070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.614736    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerStarted","Data":"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"}
Jul 01 21:03:25.615070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.615058    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:25.615209 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.615192    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:25.616436 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.616409    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:03:25.634941 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:25.634897    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podStartSLOduration=1.933636453 podStartE2EDuration="5.634883828s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.701247375s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:03:25.633806658 +0000 UTC m=+996.549084697" watchObservedRunningTime="2026-07-01 21:03:25.634883828 +0000 UTC m=+996.550161866"
Jul 01 21:03:26.543781 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:26.543744    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:03:26.544107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:26.544082    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:03:26.619399 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:26.619362    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:03:27.228272 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:27.228232    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:27.623531 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:27.623432    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:03:32.228979 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:32.228936    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:32.229403 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:32.229055    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:32.634072 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:32.633987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:03:32.634649 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:32.634613    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:03:36.544561 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:36.544514    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:03:36.545021 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:36.544923    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:03:37.228931 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:37.228890    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:42.229259 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:42.229220    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:42.635502 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:42.635422    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:03:46.543790 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:46.543746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 21:03:46.544335 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:46.543907    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:46.544335 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:46.544077    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:03:46.544335 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:46.544208    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:47.228466 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:47.228426    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 21:03:50.502905 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.502882    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:50.508545 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config\") pod \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") "
Jul 01 21:03:50.508659 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508593    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location\") pod \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") "
Jul 01 21:03:50.508659 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert\") pod \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") "
Jul 01 21:03:50.508802 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c8ch\" (UniqueName: \"kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch\") pod \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") "
Jul 01 21:03:50.508802 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508743    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls\") pod \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\" (UID: \"52b4c1eb-942a-4812-8d7d-96dd48e6635e\") "
Jul 01 21:03:50.508917 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508885    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52b4c1eb-942a-4812-8d7d-96dd48e6635e" (UID: "52b4c1eb-942a-4812-8d7d-96dd48e6635e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:03:50.509037 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508918    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52b4c1eb-942a-4812-8d7d-96dd48e6635e" (UID: "52b4c1eb-942a-4812-8d7d-96dd48e6635e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:03:50.509037 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508957    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52b4c1eb-942a-4812-8d7d-96dd48e6635e" (UID: "52b4c1eb-942a-4812-8d7d-96dd48e6635e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:03:50.509037 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.508983    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.510745 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.510687    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52b4c1eb-942a-4812-8d7d-96dd48e6635e" (UID: "52b4c1eb-942a-4812-8d7d-96dd48e6635e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:03:50.510831 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.510744    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch" (OuterVolumeSpecName: "kube-api-access-2c8ch") pod "52b4c1eb-942a-4812-8d7d-96dd48e6635e" (UID: "52b4c1eb-942a-4812-8d7d-96dd48e6635e"). InnerVolumeSpecName "kube-api-access-2c8ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:03:50.610070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.610003    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.610070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.610026    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52b4c1eb-942a-4812-8d7d-96dd48e6635e-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.610070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.610035    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8ch\" (UniqueName: \"kubernetes.io/projected/52b4c1eb-942a-4812-8d7d-96dd48e6635e-kube-api-access-2c8ch\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.610070 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.610044    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52b4c1eb-942a-4812-8d7d-96dd48e6635e-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.707960 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.707930    2574 generic.go:356] "Generic (PLEG): container finished" podID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerID="898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df" exitCode=0
Jul 01 21:03:50.708107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.707992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerDied","Data":"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"}
Jul 01 21:03:50.708107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.708044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997" event={"ID":"52b4c1eb-942a-4812-8d7d-96dd48e6635e","Type":"ContainerDied","Data":"5c264e7cac74a8da90ec03714a6e03f35cbd7a28be45c4885f185f75997d8561"}
Jul 01 21:03:50.708107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.707996    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"
Jul 01 21:03:50.708107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.708082    2574 scope.go:117] "RemoveContainer" containerID="898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"
Jul 01 21:03:50.716179 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.716162    2574 scope.go:117] "RemoveContainer" containerID="c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"
Jul 01 21:03:50.723395 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.723378    2574 scope.go:117] "RemoveContainer" containerID="d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"
Jul 01 21:03:50.730627 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.730607    2574 scope.go:117] "RemoveContainer" containerID="e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"
Jul 01 21:03:50.730814 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.730798    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:03:50.735220 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.735200    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997"]
Jul 01 21:03:50.737877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.737864    2574 scope.go:117] "RemoveContainer" containerID="898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"
Jul 01 21:03:50.738170 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:03:50.738145    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df\": container with ID starting with 898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df not found: ID does not exist" containerID="898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"
Jul 01 21:03:50.738211 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738172    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df"} err="failed to get container status \"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df\": rpc error: code = NotFound desc = could not find container \"898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df\": container with ID starting with 898517b051b689d13fdd435447311184e4ea1dcdd016eab135c3dc3d729704df not found: ID does not exist"
Jul 01 21:03:50.738211 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738189    2574 scope.go:117] "RemoveContainer" containerID="c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"
Jul 01 21:03:50.738460 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:03:50.738442    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663\": container with ID starting with c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663 not found: ID does not exist" containerID="c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"
Jul 01 21:03:50.738530 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738478    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663"} err="failed to get container status \"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663\": rpc error: code = NotFound desc = could not find container \"c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663\": container with ID starting with c54aaeffcf32b4977b7b6648e672e0c74a9bdc3ffe4be821a4f64f386fc63663 not found: ID does not exist"
Jul 01 21:03:50.738530 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738500    2574 scope.go:117] "RemoveContainer" containerID="d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"
Jul 01 21:03:50.738770 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:03:50.738751    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703\": container with ID starting with d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703 not found: ID does not exist" containerID="d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"
Jul 01 21:03:50.738864 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738773    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703"} err="failed to get container status \"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703\": rpc error: code = NotFound desc = could not find container \"d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703\": container with ID starting with d04bc397ff94d9c61aaadcd51d8d15b4456d8b7372f04b0283ba93b2a049a703 not found: ID does not exist"
Jul 01 21:03:50.738864 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.738786    2574 scope.go:117] "RemoveContainer" containerID="e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"
Jul 01 21:03:50.739051 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:03:50.739011    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c\": container with ID starting with e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c not found: ID does not exist" containerID="e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"
Jul 01 21:03:50.739125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:50.739057    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c"} err="failed to get container status \"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c\": rpc error: code = NotFound desc = could not find container \"e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c\": container with ID starting with e6717944e3af025aa659af5471943d725d53caa658b832621fe4b3490d82028c not found: ID does not exist"
Jul 01 21:03:51.548048 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:51.548019    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" path="/var/lib/kubelet/pods/52b4c1eb-942a-4812-8d7d-96dd48e6635e/volumes"
Jul 01 21:03:52.634937 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:03:52.634904    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:02.634631 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:02.634596    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:09.784347 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:09.784313    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:04:12.635323 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:12.635287    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:22.635581 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:22.635538    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:32.634857 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:32.634821    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:42.634773 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:42.634676    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:49.546119 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:49.546082    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:04:59.546048 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:04:59.545999    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:05:09.545947 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:09.545907    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:05:19.546033 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:19.545997    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:05:29.546193 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:29.546150    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:05:39.546000 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:39.545957    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:05:49.548536 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:49.548510    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:05:50.472200 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:50.472161    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:05:50.472406 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:50.472372    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" containerID="cri-o://e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1" gracePeriod=30
Jul 01 21:05:50.472543 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:50.472441    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy" containerID="cri-o://136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6" gracePeriod=30
Jul 01 21:05:50.530964 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:50.530907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:05:51.155848 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:51.155815    2574 generic.go:356] "Generic (PLEG): container finished" podID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerID="136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6" exitCode=2
Jul 01 21:05:51.155848 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:51.155850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerDied","Data":"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"}
Jul 01 21:05:52.624737 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:52.624678    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 21:05:55.630963 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.630925    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:05:55.631433 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631411    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="storage-initializer"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631436    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="storage-initializer"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631459    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631469    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631480    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631490    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container"
Jul 01 21:05:55.631510 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631503    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kserve-container"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631512    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kserve-container"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631525    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631534    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631551    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631559    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631628    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631642    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kube-rbac-proxy"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631651    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97566295-827d-4bd2-8a44-231e50c2bce0" containerName="kserve-container"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631661    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="kserve-container"
Jul 01 21:05:55.631842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.631670    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b4c1eb-942a-4812-8d7d-96dd48e6635e" containerName="agent"
Jul 01 21:05:55.635017 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.634998    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.640164 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.640139    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-77d927-kube-rbac-proxy-sar-config\""
Jul 01 21:05:55.640286 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.640187    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-77d927-predictor-serving-cert\""
Jul 01 21:05:55.649639 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.649617    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:05:55.671133 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.671103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:05:55.722471 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.722430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.722471 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.722481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.722708 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.722546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpk4\" (UniqueName: \"kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.722708 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.722591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.722708 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.722675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.763562 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.763529    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:05:55.823502 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.823470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.823658 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.823508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.823658 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.823553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.823658 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.823579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpk4\" (UniqueName: \"kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.823658 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.823598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.824050 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.824022    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.824325 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.824303    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.824377 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.824303    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.826165 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.826136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.832451 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.832431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpk4\" (UniqueName: \"kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4\") pod \"isvc-primary-77d927-predictor-86f44cb6d7-qvq7f\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") " pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:55.944686 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:55.944656    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:05:56.092601 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:56.092564    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:05:56.096458 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:05:56.096418    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660ad75b_aeb2_4663_ab29_a21312b52718.slice/crio-fdab6f9d63e205830a8fbfa551f75f1ccc71f892e241fa260bb3c4d0ff7eb176 WatchSource:0}: Error finding container fdab6f9d63e205830a8fbfa551f75f1ccc71f892e241fa260bb3c4d0ff7eb176: Status 404 returned error can't find the container with id fdab6f9d63e205830a8fbfa551f75f1ccc71f892e241fa260bb3c4d0ff7eb176
Jul 01 21:05:56.176941 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:56.176910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerStarted","Data":"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"}
Jul 01 21:05:56.177075 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:56.176983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerStarted","Data":"fdab6f9d63e205830a8fbfa551f75f1ccc71f892e241fa260bb3c4d0ff7eb176"}
Jul 01 21:05:57.624250 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:57.624212    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 21:05:59.546108 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:05:59.546055    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 21:06:00.120134 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.120110    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:06:00.193652 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.193618    2574 generic.go:356] "Generic (PLEG): container finished" podID="660ad75b-aeb2-4663-ab29-a21312b52718" containerID="3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19" exitCode=0
Jul 01 21:06:00.193842 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.193692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerDied","Data":"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"}
Jul 01 21:06:00.200262 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.200228    2574 generic.go:356] "Generic (PLEG): container finished" podID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerID="e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1" exitCode=0
Jul 01 21:06:00.200414 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.200290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerDied","Data":"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"}
Jul 01 21:06:00.200414 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.200316    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"
Jul 01 21:06:00.200618 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.200411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2" event={"ID":"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6","Type":"ContainerDied","Data":"ddf23ae0bce8b12f8ce4906d2f676f06c57a0dc761680bee97774b9dd41b5b17"}
Jul 01 21:06:00.200618 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.200454    2574 scope.go:117] "RemoveContainer" containerID="136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"
Jul 01 21:06:00.210950 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.210930    2574 scope.go:117] "RemoveContainer" containerID="e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"
Jul 01 21:06:00.222302 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.222278    2574 scope.go:117] "RemoveContainer" containerID="2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"
Jul 01 21:06:00.236052 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236034    2574 scope.go:117] "RemoveContainer" containerID="136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"
Jul 01 21:06:00.236423 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:06:00.236401    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6\": container with ID starting with 136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6 not found: ID does not exist" containerID="136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"
Jul 01 21:06:00.236517 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236431    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6"} err="failed to get container status \"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6\": rpc error: code = NotFound desc = could not find container \"136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6\": container with ID starting with 136d0e8b5fcf20836e871e9d87ca991bae2a6bc225ea4b2eb7b9005c4cd305d6 not found: ID does not exist"
Jul 01 21:06:00.236517 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236451    2574 scope.go:117] "RemoveContainer" containerID="e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"
Jul 01 21:06:00.236743 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:06:00.236700    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1\": container with ID starting with e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1 not found: ID does not exist" containerID="e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"
Jul 01 21:06:00.236816 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236745    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1"} err="failed to get container status \"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1\": rpc error: code = NotFound desc = could not find container \"e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1\": container with ID starting with e0579857d7db16221649ede54c5096167e56a85203286a50819db09234481ff1 not found: ID does not exist"
Jul 01 21:06:00.236816 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236761    2574 scope.go:117] "RemoveContainer" containerID="2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"
Jul 01 21:06:00.236995 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:06:00.236978    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db\": container with ID starting with 2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db not found: ID does not exist" containerID="2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"
Jul 01 21:06:00.237046 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.236999    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db"} err="failed to get container status \"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db\": rpc error: code = NotFound desc = could not find container \"2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db\": container with ID starting with 2ffbd0e15b8418ec9bcb553aa89c0be7d5b933a066ee0a7646bd0fb9deeb03db not found: ID does not exist"
Jul 01 21:06:00.258269 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snzkl\" (UniqueName: \"kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl\") pod \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") "
Jul 01 21:06:00.258379 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258279    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert\") pod \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") "
Jul 01 21:06:00.258379 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258300    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config\") pod \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") "
Jul 01 21:06:00.258589 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258465    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location\") pod \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") "
Jul 01 21:06:00.258668 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258610    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls\") pod \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\" (UID: \"db7dc66b-dfc8-48d3-8e10-024fc08f7cd6\") "
Jul 01 21:06:00.258799 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258742    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" (UID: "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:00.258799 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.258776    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" (UID: "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:06:00.259036 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.259021    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.259116 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.259041    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.259203 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.259179    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" (UID: "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:00.260656 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.260633    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl" (OuterVolumeSpecName: "kube-api-access-snzkl") pod "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" (UID: "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6"). InnerVolumeSpecName "kube-api-access-snzkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:06:00.260945 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.260924    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" (UID: "db7dc66b-dfc8-48d3-8e10-024fc08f7cd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:06:00.359790 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.359676    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.359790 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.359708    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snzkl\" (UniqueName: \"kubernetes.io/projected/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-api-access-snzkl\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.359790 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.359740    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.523436 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.523398    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:06:00.525234 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:00.525213    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2"]
Jul 01 21:06:01.206836 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.206802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerStarted","Data":"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"}
Jul 01 21:06:01.207308 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.206885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerStarted","Data":"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"}
Jul 01 21:06:01.207308 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.207201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:06:01.207308 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.207256    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:06:01.208391 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.208360    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:01.226053 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.225999    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podStartSLOduration=2.531826762 podStartE2EDuration="6.225979834s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.694153072s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:06:01.225174558 +0000 UTC m=+1152.140452622" watchObservedRunningTime="2026-07-01 21:06:01.225979834 +0000 UTC m=+1152.141257917"
Jul 01 21:06:01.548472 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:01.548399    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" path="/var/lib/kubelet/pods/db7dc66b-dfc8-48d3-8e10-024fc08f7cd6/volumes"
Jul 01 21:06:02.212659 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:02.212618    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:07.217708 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:07.217678    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:06:07.218258 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:07.218231    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:17.218613 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:17.218576    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:27.218547 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:27.218506    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:37.218131 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:37.218091    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:47.218320 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:47.218279    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:06:49.577505 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:49.577479    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:06:49.578886 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:49.578865    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:06:57.218281 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:06:57.218237    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:07:07.218253 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:07.218216    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 21:07:11.553930 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:11.553901    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:07:15.682704 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.682669    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:15.706857 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.706829    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:15.707146 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707128    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy"
Jul 01 21:07:15.707146 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707141    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707161    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="storage-initializer"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707166    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="storage-initializer"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707172    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707178    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707222    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kserve-container"
Jul 01 21:07:15.707241 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.707231    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7dc66b-dfc8-48d3-8e10-024fc08f7cd6" containerName="kube-rbac-proxy"
Jul 01 21:07:15.710281 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.710264    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.713312 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.713290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-77d927-predictor-serving-cert\""
Jul 01 21:07:15.713425 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.713361    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-77d927\""
Jul 01 21:07:15.713425 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.713385    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-77d927-dockercfg-7h84v\""
Jul 01 21:07:15.713425 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.713395    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-77d927-kube-rbac-proxy-sar-config\""
Jul 01 21:07:15.720911 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.720889    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:15.852674 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.852648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.852877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.852683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.852877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.852733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzpl\" (UniqueName: \"kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.852877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.852752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.852877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.852834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.867513 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.867485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:15.954148 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954316 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954316 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzpl\" (UniqueName: \"kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954316 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954316 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954698 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954674    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.954997 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.954978    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.955069 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.955017    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.956621 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.956603    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:15.962877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:15.962855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzpl\" (UniqueName: \"kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl\") pod \"isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") " pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:16.019832 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:16.019790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:16.155838 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:16.155798    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:16.161360 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:07:16.161333    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2881f6d_1c8b_4dcf_89b8_bc38d9de1f02.slice/crio-38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11 WatchSource:0}: Error finding container 38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11: Status 404 returned error can't find the container with id 38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11
Jul 01 21:07:16.501660 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:16.501571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerStarted","Data":"ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"}
Jul 01 21:07:16.501829 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:16.501670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerStarted","Data":"38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11"}
Jul 01 21:07:21.524049 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:21.524024    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/0.log"
Jul 01 21:07:21.524443 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:21.524063    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerID="ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a" exitCode=1
Jul 01 21:07:21.524443 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:21.524095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerDied","Data":"ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"}
Jul 01 21:07:22.529199 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:22.529169    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/0.log"
Jul 01 21:07:22.529579 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:22.529250    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerStarted","Data":"e395941e29e96bd61920585059574f8cbb459547ffc9f22a870b678cc7f56d8d"}
Jul 01 21:07:27.550323 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.550294    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/1.log"
Jul 01 21:07:27.550732 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.550623    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/0.log"
Jul 01 21:07:27.550732 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.550654    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerID="e395941e29e96bd61920585059574f8cbb459547ffc9f22a870b678cc7f56d8d" exitCode=1
Jul 01 21:07:27.550732 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.550691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerDied","Data":"e395941e29e96bd61920585059574f8cbb459547ffc9f22a870b678cc7f56d8d"}
Jul 01 21:07:27.550860 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.550762    2574 scope.go:117] "RemoveContainer" containerID="ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"
Jul 01 21:07:27.551101 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.551084    2574 scope.go:117] "RemoveContainer" containerID="ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"
Jul 01 21:07:27.560979 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:27.560948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_kserve-ci-e2e-test(f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02)\"" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"
Jul 01 21:07:27.561081 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:27.561032    2574 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_kserve-ci-e2e-test_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02_0 in pod sandbox 38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11 from index: no such id: 'ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a'" containerID="ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"
Jul 01 21:07:27.561152 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:27.561078    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_kserve-ci-e2e-test_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02_0 in pod sandbox 38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11 from index: no such id: 'ae2a78fa135d98aaa429da75349283fda56da3092385cefd0ac16e245b5c2d4a'"
Jul 01 21:07:28.556095 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:28.556072    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/1.log"
Jul 01 21:07:33.786741 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.786691    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:33.812043 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.812011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:33.836415 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.836386    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:07:33.836676 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.836652    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container" containerID="cri-o://91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1" gracePeriod=30
Jul 01 21:07:33.836800 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.836700    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kube-rbac-proxy" containerID="cri-o://49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2" gracePeriod=30
Jul 01 21:07:33.885997 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.885972    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:33.955513 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.955494    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/1.log"
Jul 01 21:07:33.957607 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:33.957591    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:34.100252 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100186    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location\") pod \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") "
Jul 01 21:07:34.100252 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100227    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpzpl\" (UniqueName: \"kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl\") pod \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") "
Jul 01 21:07:34.100435 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert\") pod \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") "
Jul 01 21:07:34.100435 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls\") pod \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") "
Jul 01 21:07:34.100435 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100315    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config\") pod \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\" (UID: \"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02\") "
Jul 01 21:07:34.100574 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100441    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" (UID: "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:07:34.100626 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100576    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:34.100696 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100642    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" (UID: "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:34.100696 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.100680    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" (UID: "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:34.102303 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.102279    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl" (OuterVolumeSpecName: "kube-api-access-qpzpl") pod "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" (UID: "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"). InnerVolumeSpecName "kube-api-access-qpzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:34.102424 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.102406    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" (UID: "f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:34.200905 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.200878    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzpl\" (UniqueName: \"kubernetes.io/projected/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-api-access-qpzpl\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:34.200905 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.200901    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:34.200905 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.200912    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:34.201090 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.200920    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:34.578914 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.578887    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/storage-initializer/1.log"
Jul 01 21:07:34.579058 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.578989    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"
Jul 01 21:07:34.579058 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.579024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf" event={"ID":"f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02","Type":"ContainerDied","Data":"38e0d3a22f4caca6785723f38eebd5ed421782f09fac7361094012aa6fe03a11"}
Jul 01 21:07:34.579176 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.579107    2574 scope.go:117] "RemoveContainer" containerID="e395941e29e96bd61920585059574f8cbb459547ffc9f22a870b678cc7f56d8d"
Jul 01 21:07:34.581459 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.581434    2574 generic.go:356] "Generic (PLEG): container finished" podID="660ad75b-aeb2-4663-ab29-a21312b52718" containerID="49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2" exitCode=2
Jul 01 21:07:34.581590 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.581490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerDied","Data":"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"}
Jul 01 21:07:34.623881 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.623859    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:34.628940 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:34.628916    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf"]
Jul 01 21:07:35.552466 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:35.552428    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" path="/var/lib/kubelet/pods/f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02/volumes"
Jul 01 21:07:37.213618 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:37.213573    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 01 21:07:38.371789 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.371769    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:07:38.532536 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532452    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config\") pod \"660ad75b-aeb2-4663-ab29-a21312b52718\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") "
Jul 01 21:07:38.532536 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location\") pod \"660ad75b-aeb2-4663-ab29-a21312b52718\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") "
Jul 01 21:07:38.532536 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532527    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls\") pod \"660ad75b-aeb2-4663-ab29-a21312b52718\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") "
Jul 01 21:07:38.532826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert\") pod \"660ad75b-aeb2-4663-ab29-a21312b52718\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") "
Jul 01 21:07:38.532826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jpk4\" (UniqueName: \"kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4\") pod \"660ad75b-aeb2-4663-ab29-a21312b52718\" (UID: \"660ad75b-aeb2-4663-ab29-a21312b52718\") "
Jul 01 21:07:38.532932 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532862    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "660ad75b-aeb2-4663-ab29-a21312b52718" (UID: "660ad75b-aeb2-4663-ab29-a21312b52718"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:07:38.532932 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532911    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "660ad75b-aeb2-4663-ab29-a21312b52718" (UID: "660ad75b-aeb2-4663-ab29-a21312b52718"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:38.532932 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.532923    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "660ad75b-aeb2-4663-ab29-a21312b52718" (UID: "660ad75b-aeb2-4663-ab29-a21312b52718"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:38.534918 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.534895    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "660ad75b-aeb2-4663-ab29-a21312b52718" (UID: "660ad75b-aeb2-4663-ab29-a21312b52718"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:38.535268 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.534914    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4" (OuterVolumeSpecName: "kube-api-access-9jpk4") pod "660ad75b-aeb2-4663-ab29-a21312b52718" (UID: "660ad75b-aeb2-4663-ab29-a21312b52718"). InnerVolumeSpecName "kube-api-access-9jpk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:38.600108 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.600083    2574 generic.go:356] "Generic (PLEG): container finished" podID="660ad75b-aeb2-4663-ab29-a21312b52718" containerID="91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1" exitCode=0
Jul 01 21:07:38.600191 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.600135    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"
Jul 01 21:07:38.600191 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.600153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerDied","Data":"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"}
Jul 01 21:07:38.600268 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.600207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f" event={"ID":"660ad75b-aeb2-4663-ab29-a21312b52718","Type":"ContainerDied","Data":"fdab6f9d63e205830a8fbfa551f75f1ccc71f892e241fa260bb3c4d0ff7eb176"}
Jul 01 21:07:38.600268 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.600231    2574 scope.go:117] "RemoveContainer" containerID="49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"
Jul 01 21:07:38.608837 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.608823    2574 scope.go:117] "RemoveContainer" containerID="91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"
Jul 01 21:07:38.615766 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.615751    2574 scope.go:117] "RemoveContainer" containerID="3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"
Jul 01 21:07:38.622973 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.622954    2574 scope.go:117] "RemoveContainer" containerID="49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"
Jul 01 21:07:38.623052 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623034    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:07:38.623233 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:38.623217    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2\": container with ID starting with 49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2 not found: ID does not exist" containerID="49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"
Jul 01 21:07:38.623272 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623241    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2"} err="failed to get container status \"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2\": rpc error: code = NotFound desc = could not find container \"49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2\": container with ID starting with 49ca194f8ca55d3b851fdaa5ea83b640c99a4a5ebd416efa64cdbce0a2ba2af2 not found: ID does not exist"
Jul 01 21:07:38.623272 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623259    2574 scope.go:117] "RemoveContainer" containerID="91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"
Jul 01 21:07:38.623466 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:38.623447    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1\": container with ID starting with 91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1 not found: ID does not exist" containerID="91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"
Jul 01 21:07:38.623554 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623469    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1"} err="failed to get container status \"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1\": rpc error: code = NotFound desc = could not find container \"91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1\": container with ID starting with 91fa5ca71c711236ab7d3fb4975dc1d0b075f172650d76217654ab4793d295f1 not found: ID does not exist"
Jul 01 21:07:38.623554 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623484    2574 scope.go:117] "RemoveContainer" containerID="3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"
Jul 01 21:07:38.623680 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:38.623662    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19\": container with ID starting with 3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19 not found: ID does not exist" containerID="3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"
Jul 01 21:07:38.623732 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.623685    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19"} err="failed to get container status \"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19\": rpc error: code = NotFound desc = could not find container \"3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19\": container with ID starting with 3ec2e9e18e29b04ce9afdd23c2cdd16953395fcb2eb9c77602ed0e6ca792fa19 not found: ID does not exist"
Jul 01 21:07:38.627215 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.627197    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f"]
Jul 01 21:07:38.633954 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.633934    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jpk4\" (UniqueName: \"kubernetes.io/projected/660ad75b-aeb2-4663-ab29-a21312b52718-kube-api-access-9jpk4\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:38.633954 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.633953    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:38.634067 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.633963    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660ad75b-aeb2-4663-ab29-a21312b52718-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:38.634067 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.633974    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660ad75b-aeb2-4663-ab29-a21312b52718-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:38.634067 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.633982    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660ad75b-aeb2-4663-ab29-a21312b52718-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:38.963853 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.963826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:38.993769 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.993744    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:38.994040 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994027    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994084 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994042    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994084 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994059    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="storage-initializer"
Jul 01 21:07:38.994084 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994066    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="storage-initializer"
Jul 01 21:07:38.994084 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994072    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container"
Jul 01 21:07:38.994084 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994079    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994089    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994094    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994100    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kube-rbac-proxy"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994105    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kube-rbac-proxy"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994155    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kserve-container"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994163    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994168    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02" containerName="storage-initializer"
Jul 01 21:07:38.994228 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.994174    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" containerName="kube-rbac-proxy"
Jul 01 21:07:38.998502 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:38.998486    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.001051 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001034    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:07:39.001341 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001316    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:07:39.001455 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001319    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2dd56b-predictor-serving-cert\""
Jul 01 21:07:39.001527 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001333    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2dd56b-kube-rbac-proxy-sar-config\""
Jul 01 21:07:39.001527 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001336    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:07:39.001615 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001355    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2dd56b\""
Jul 01 21:07:39.001759 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.001746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2dd56b-dockercfg-v9dgs\""
Jul 01 21:07:39.007540 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.007520    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:39.137473 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.137443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfcl\" (UniqueName: \"kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.137598 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.137494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.137598 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.137526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.137598 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.137561    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.137771 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.137616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.171836 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.171790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:39.238003 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.237949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfcl\" (UniqueName: \"kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238003 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.237988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238122 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238122 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238122 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238460 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238440    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238634 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.238702 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.238683    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.240605 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.240578    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.246764 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.246744    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfcl\" (UniqueName: \"kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl\") pod \"isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.309617 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.309585    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:39.442274 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.442250    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:39.444746 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:07:39.444701    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bf6eed_7619_4c20_b1c1_78d48e3c6e5b.slice/crio-0f1cce0db159dbcc306d607f9596a479d42524819c3acf108cc5b01cae54456a WatchSource:0}: Error finding container 0f1cce0db159dbcc306d607f9596a479d42524819c3acf108cc5b01cae54456a: Status 404 returned error can't find the container with id 0f1cce0db159dbcc306d607f9596a479d42524819c3acf108cc5b01cae54456a
Jul 01 21:07:39.547632 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.547590    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660ad75b-aeb2-4663-ab29-a21312b52718" path="/var/lib/kubelet/pods/660ad75b-aeb2-4663-ab29-a21312b52718/volumes"
Jul 01 21:07:39.612599 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.612567    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerStarted","Data":"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"}
Jul 01 21:07:39.612763 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:39.612639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerStarted","Data":"0f1cce0db159dbcc306d607f9596a479d42524819c3acf108cc5b01cae54456a"}
Jul 01 21:07:44.635286 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:44.635254    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/0.log"
Jul 01 21:07:44.635806 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:44.635304    2574 generic.go:356] "Generic (PLEG): container finished" podID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerID="78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e" exitCode=1
Jul 01 21:07:44.635806 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:44.635387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerDied","Data":"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"}
Jul 01 21:07:45.640750 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:45.640700    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/0.log"
Jul 01 21:07:45.641205 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:45.640770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerStarted","Data":"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"}
Jul 01 21:07:48.959987 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:48.959958    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:48.960364 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:48.960186    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer" containerID="cri-o://861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975" gracePeriod=30
Jul 01 21:07:48.979839 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:48.979814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:51.397180 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.397158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/1.log"
Jul 01 21:07:51.397577 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.397560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/0.log"
Jul 01 21:07:51.399808 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.399794    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:51.431382 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431363    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcfcl\" (UniqueName: \"kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl\") pod \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") "
Jul 01 21:07:51.431514 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431399    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls\") pod \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") "
Jul 01 21:07:51.431514 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431448    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location\") pod \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") "
Jul 01 21:07:51.431514 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert\") pod \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") "
Jul 01 21:07:51.431682 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config\") pod \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\" (UID: \"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b\") "
Jul 01 21:07:51.431777 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431751    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" (UID: "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:07:51.431967 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431940    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" (UID: "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:51.431967 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.431944    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" (UID: "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:51.433499 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.433481    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl" (OuterVolumeSpecName: "kube-api-access-pcfcl") pod "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" (UID: "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b"). InnerVolumeSpecName "kube-api-access-pcfcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:51.433585 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.433507    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" (UID: "f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:51.532630 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.532564    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfcl\" (UniqueName: \"kubernetes.io/projected/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-api-access-pcfcl\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:51.532630 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.532595    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:51.532630 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.532605    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:51.532630 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.532614    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:51.532630 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.532623    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:07:51.665240 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665212    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/1.log"
Jul 01 21:07:51.665608 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665592    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds_f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/storage-initializer/0.log"
Jul 01 21:07:51.665677 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665632    2574 generic.go:356] "Generic (PLEG): container finished" podID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerID="861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975" exitCode=1
Jul 01 21:07:51.665754 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerDied","Data":"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"}
Jul 01 21:07:51.665754 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665729    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"
Jul 01 21:07:51.665865 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds" event={"ID":"f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b","Type":"ContainerDied","Data":"0f1cce0db159dbcc306d607f9596a479d42524819c3acf108cc5b01cae54456a"}
Jul 01 21:07:51.665865 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.665821    2574 scope.go:117] "RemoveContainer" containerID="861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"
Jul 01 21:07:51.676076 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.676052    2574 scope.go:117] "RemoveContainer" containerID="78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"
Jul 01 21:07:51.683430 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.683416    2574 scope.go:117] "RemoveContainer" containerID="861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"
Jul 01 21:07:51.683706 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:51.683688    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975\": container with ID starting with 861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975 not found: ID does not exist" containerID="861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"
Jul 01 21:07:51.683778 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.683731    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975"} err="failed to get container status \"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975\": rpc error: code = NotFound desc = could not find container \"861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975\": container with ID starting with 861fc85f91d5a099f1b09d93f13908d70e1b87ad84b3252a3266704127b40975 not found: ID does not exist"
Jul 01 21:07:51.683778 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.683747    2574 scope.go:117] "RemoveContainer" containerID="78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"
Jul 01 21:07:51.683975 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:51.683957    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e\": container with ID starting with 78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e not found: ID does not exist" containerID="78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"
Jul 01 21:07:51.684012 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.683983    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e"} err="failed to get container status \"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e\": rpc error: code = NotFound desc = could not find container \"78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e\": container with ID starting with 78ffc7ce5199d5b7f3f9fa5c6a4982b527d6cae7d2c89f9fcb6283fc87ac937e not found: ID does not exist"
Jul 01 21:07:51.698454 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.698435    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:51.704357 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:51.704337    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds"]
Jul 01 21:07:53.547921 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:53.547880    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" path="/var/lib/kubelet/pods/f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b/volumes"
Jul 01 21:07:54.215728 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.215679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:54.277173 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277141    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:07:54.277490 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277473    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.277562 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277493    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.277627 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.277627 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277597    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.277767 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277675    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.277767 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.277686    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3bf6eed-7619-4c20-b1c1-78d48e3c6e5b" containerName="storage-initializer"
Jul 01 21:07:54.282266 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.282246    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.285123 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.285102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-92cf8-predictor-serving-cert\""
Jul 01 21:07:54.285242 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.285184    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:07:54.285662 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.285643    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-92cf8-kube-rbac-proxy-sar-config\""
Jul 01 21:07:54.285662 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.285655    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7bgz\""
Jul 01 21:07:54.285824 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.285768    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:07:54.286104 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.286086    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:07:54.286301 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.286285    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:07:54.297920 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.297899    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:07:54.350508 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.350489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zh4\" (UniqueName: \"kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.350629 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.350519    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.350629 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.350538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.350629 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.350614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.350787 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.350651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451073 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zh4\" (UniqueName: \"kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451096    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451399 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:54.451217    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-serving-cert: secret "raw-sklearn-92cf8-predictor-serving-cert" not found
Jul 01 21:07:54.451399 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:07:54.451275    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls podName:d3ebb858-b3ad-44ca-9ee6-ef6508714873 nodeName:}" failed. No retries permitted until 2026-07-01 21:07:54.951258882 +0000 UTC m=+1265.866536899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls") pod "raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873") : secret "raw-sklearn-92cf8-predictor-serving-cert" not found
Jul 01 21:07:54.451593 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451575    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451796 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451780    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.451876 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.451860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.475902 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.475847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zh4\" (UniqueName: \"kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.665225 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.665196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:54.954810 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.954790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:54.957201 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:54.957175    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") pod \"raw-sklearn-92cf8-predictor-d4757fc6b-s49vs\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") " pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:55.192190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:55.192165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:07:55.317470 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:55.317428    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:07:55.319503 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:07:55.319465    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ebb858_b3ad_44ca_9ee6_ef6508714873.slice/crio-c663a73e6bb749512d6c709bc0f568bd38cecfdb7e80b85e1b948da87eed35f2 WatchSource:0}: Error finding container c663a73e6bb749512d6c709bc0f568bd38cecfdb7e80b85e1b948da87eed35f2: Status 404 returned error can't find the container with id c663a73e6bb749512d6c709bc0f568bd38cecfdb7e80b85e1b948da87eed35f2
Jul 01 21:07:55.683126 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:55.683047    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerStarted","Data":"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"}
Jul 01 21:07:55.683126 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:55.683102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerStarted","Data":"c663a73e6bb749512d6c709bc0f568bd38cecfdb7e80b85e1b948da87eed35f2"}
Jul 01 21:07:56.420449 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:56.420418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:07:59.700953 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:59.700921    2574 generic.go:356] "Generic (PLEG): container finished" podID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerID="d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f" exitCode=0
Jul 01 21:07:59.701278 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:07:59.700959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerDied","Data":"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"}
Jul 01 21:08:00.707132 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:00.707096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerStarted","Data":"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"}
Jul 01 21:08:00.707482 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:00.707161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerStarted","Data":"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"}
Jul 01 21:08:00.707482 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:00.707345    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:08:00.726056 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:00.725984    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podStartSLOduration=2.929378842 podStartE2EDuration="6.725970652s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79659181s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:08:00.725941534 +0000 UTC m=+1271.641219591" watchObservedRunningTime="2026-07-01 21:08:00.725970652 +0000 UTC m=+1271.641248691"
Jul 01 21:08:01.711320 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:01.711283    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:08:01.712637 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:01.712609    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:02.714856 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:02.714817    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:07.718978 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:07.718949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:08:07.719455 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:07.719428    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:12.135077 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:12.135046    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:08:17.719461 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:17.719418    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:27.719580 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:27.719542    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:37.719962 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:37.719920    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:47.720381 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:47.720344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:08:57.719494 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:08:57.719447    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:09:07.719998 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:07.719971    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:09:14.231460 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.231425    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:09:14.231886 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.231668    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kube-rbac-proxy" containerID="cri-o://ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c" gracePeriod=30
Jul 01 21:09:14.231886 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.231650    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" containerID="cri-o://9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7" gracePeriod=30
Jul 01 21:09:14.254032 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.254006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:09:14.988589 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.988557    2574 generic.go:356] "Generic (PLEG): container finished" podID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerID="ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c" exitCode=2
Jul 01 21:09:14.988795 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:14.988604    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerDied","Data":"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"}
Jul 01 21:09:16.326685 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:16.326643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:09:17.715457 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:17.715420    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 21:09:17.719786 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:17.719763    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 21:09:18.701285 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.701261    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:09:18.779991 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.779911    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5zh4\" (UniqueName: \"kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4\") pod \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") "
Jul 01 21:09:18.779991 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.779946    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert\") pod \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") "
Jul 01 21:09:18.779991 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.779972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") pod \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") "
Jul 01 21:09:18.780525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.780016    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config\") pod \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") "
Jul 01 21:09:18.780525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.780037    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location\") pod \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\" (UID: \"d3ebb858-b3ad-44ca-9ee6-ef6508714873\") "
Jul 01 21:09:18.780525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.780422    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3ebb858-b3ad-44ca-9ee6-ef6508714873" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:18.780525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.780434    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3ebb858-b3ad-44ca-9ee6-ef6508714873" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:18.780525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.780459    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3ebb858-b3ad-44ca-9ee6-ef6508714873" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:09:18.782107 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.782081    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3ebb858-b3ad-44ca-9ee6-ef6508714873" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:09:18.782208 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.782138    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4" (OuterVolumeSpecName: "kube-api-access-n5zh4") pod "d3ebb858-b3ad-44ca-9ee6-ef6508714873" (UID: "d3ebb858-b3ad-44ca-9ee6-ef6508714873"). InnerVolumeSpecName "kube-api-access-n5zh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:09:18.881002 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.880975    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zh4\" (UniqueName: \"kubernetes.io/projected/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-api-access-n5zh4\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:09:18.881002 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.881001    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:09:18.881002 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.881012    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ebb858-b3ad-44ca-9ee6-ef6508714873-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:09:18.881221 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.881020    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:09:18.881221 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:18.881029    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ebb858-b3ad-44ca-9ee6-ef6508714873-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:09:19.006104 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.006073    2574 generic.go:356] "Generic (PLEG): container finished" podID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerID="9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7" exitCode=0
Jul 01 21:09:19.006271 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.006110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerDied","Data":"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"}
Jul 01 21:09:19.006271 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.006132    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"
Jul 01 21:09:19.006271 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.006157    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs" event={"ID":"d3ebb858-b3ad-44ca-9ee6-ef6508714873","Type":"ContainerDied","Data":"c663a73e6bb749512d6c709bc0f568bd38cecfdb7e80b85e1b948da87eed35f2"}
Jul 01 21:09:19.006271 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.006184    2574 scope.go:117] "RemoveContainer" containerID="ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"
Jul 01 21:09:19.020234 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.020216    2574 scope.go:117] "RemoveContainer" containerID="9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"
Jul 01 21:09:19.027317 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.027297    2574 scope.go:117] "RemoveContainer" containerID="d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"
Jul 01 21:09:19.032463 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.032414    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:09:19.034548 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.034532    2574 scope.go:117] "RemoveContainer" containerID="ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"
Jul 01 21:09:19.034835 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:09:19.034815    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c\": container with ID starting with ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c not found: ID does not exist" containerID="ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"
Jul 01 21:09:19.034896 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.034842    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c"} err="failed to get container status \"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c\": rpc error: code = NotFound desc = could not find container \"ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c\": container with ID starting with ef6e2b9c29452f7e4981a5fe5401577ce7dab303c7ea574023abe1bda21c122c not found: ID does not exist"
Jul 01 21:09:19.034896 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.034859    2574 scope.go:117] "RemoveContainer" containerID="9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"
Jul 01 21:09:19.035097 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:09:19.035078    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7\": container with ID starting with 9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7 not found: ID does not exist" containerID="9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"
Jul 01 21:09:19.035179 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.035103    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7"} err="failed to get container status \"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7\": rpc error: code = NotFound desc = could not find container \"9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7\": container with ID starting with 9d7ed40144da887a8570c876595b8056a261a0550fdbd1d2f778145c7242d6d7 not found: ID does not exist"
Jul 01 21:09:19.035179 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.035123    2574 scope.go:117] "RemoveContainer" containerID="d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"
Jul 01 21:09:19.035768 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:09:19.035703    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f\": container with ID starting with d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f not found: ID does not exist" containerID="d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"
Jul 01 21:09:19.035868 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.035777    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f"} err="failed to get container status \"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f\": rpc error: code = NotFound desc = could not find container \"d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f\": container with ID starting with d4d3e71df0430975c3d16216f5ebaa83ceb9a6bcbefb61b5b188dd2fef7faf4f not found: ID does not exist"
Jul 01 21:09:19.036959 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.036941    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs"]
Jul 01 21:09:19.359737 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.359642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:09:19.405328 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405301    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:09:19.405627 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405614    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container"
Jul 01 21:09:19.405670 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405628    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container"
Jul 01 21:09:19.405670 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405637    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="storage-initializer"
Jul 01 21:09:19.405670 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405655    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="storage-initializer"
Jul 01 21:09:19.405670 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405665    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kube-rbac-proxy"
Jul 01 21:09:19.405670 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405671    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kube-rbac-proxy"
Jul 01 21:09:19.405866 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405730    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kserve-container"
Jul 01 21:09:19.405866 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.405738    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" containerName="kube-rbac-proxy"
Jul 01 21:09:19.409920 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.409905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.413309 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.413289    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:09:19.413849 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.413830    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z7bgz\""
Jul 01 21:09:19.413949 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.413830    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:09:19.414123 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.414106    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:09:19.414745 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.414725    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-830cb-predictor-serving-cert\""
Jul 01 21:09:19.414843 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.414750    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-830cb-kube-rbac-proxy-sar-config\""
Jul 01 21:09:19.414843 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.414729    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:09:19.421588 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.421569    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:09:19.485320 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.485288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.485466 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.485325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.485466 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.485353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.485543 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.485456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.485543 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.485523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpq8\" (UniqueName: \"kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.548020 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.547989    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ebb858-b3ad-44ca-9ee6-ef6508714873" path="/var/lib/kubelet/pods/d3ebb858-b3ad-44ca-9ee6-ef6508714873/volumes"
Jul 01 21:09:19.586573 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.586540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpq8\" (UniqueName: \"kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.586573 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.586573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.586823 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.586602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.586823 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.586640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.586823 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.586688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.587110 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.587091    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.587386 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.587367    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.587439 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.587380    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.589171 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.589153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.596014 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.595992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpq8\" (UniqueName: \"kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8\") pod \"raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.622188 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.622109    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:09:19.720433 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.720404    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:19.849007 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:19.848947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:09:19.851619 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:09:19.851578    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28adbd4b_9edd_49bf_aede_1ad7f0e0366f.slice/crio-8ee1ffbfef04388553627f650166f749df710786ba64f24f0250b9416350f26e WatchSource:0}: Error finding container 8ee1ffbfef04388553627f650166f749df710786ba64f24f0250b9416350f26e: Status 404 returned error can't find the container with id 8ee1ffbfef04388553627f650166f749df710786ba64f24f0250b9416350f26e
Jul 01 21:09:20.011299 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:20.011249    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerStarted","Data":"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"}
Jul 01 21:09:20.011476 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:20.011329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerStarted","Data":"8ee1ffbfef04388553627f650166f749df710786ba64f24f0250b9416350f26e"}
Jul 01 21:09:24.028663 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:24.028624    2574 generic.go:356] "Generic (PLEG): container finished" podID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerID="453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25" exitCode=0
Jul 01 21:09:24.029188 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:24.028684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerDied","Data":"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"}
Jul 01 21:09:25.035329 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.035297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerStarted","Data":"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"}
Jul 01 21:09:25.035742 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.035359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerStarted","Data":"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"}
Jul 01 21:09:25.035742 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.035645    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:25.035742 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.035703    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:25.037044 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.037018    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:09:25.056229 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:25.056191    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podStartSLOduration=2.51465053 podStartE2EDuration="6.056179662s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.541529132s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:09:25.054400624 +0000 UTC m=+1355.969678660" watchObservedRunningTime="2026-07-01 21:09:25.056179662 +0000 UTC m=+1355.971457678"
Jul 01 21:09:26.038929 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:26.038885    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:09:31.043813 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:31.043784    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:09:31.044358 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:31.044332    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:09:41.044599 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:41.044562    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:09:51.044828 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:09:51.044788    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:01.044650 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:01.044609    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:11.045177 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:11.045132    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:21.044394 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:21.044350    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:31.044942 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:31.044901    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:41.044999 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:41.044913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:10:49.549029 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:49.549002    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:10:49.549481 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:49.549206    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" containerID="cri-o://73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992" gracePeriod=30
Jul 01 21:10:49.549481 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:49.549210    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kube-rbac-proxy" containerID="cri-o://e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15" gracePeriod=30
Jul 01 21:10:49.568804 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:49.568778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:10:50.362203 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:50.362171    2574 generic.go:356] "Generic (PLEG): container finished" podID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerID="e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15" exitCode=2
Jul 01 21:10:50.362392 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:50.362215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerDied","Data":"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"}
Jul 01 21:10:51.040097 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:51.040057    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 01 21:10:51.044384 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:51.044355    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 21:10:54.001546 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.001522    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:10:54.150015 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.149940    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location\") pod \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") "
Jul 01 21:10:54.150015 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.149985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls\") pod \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") "
Jul 01 21:10:54.150224 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150019    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert\") pod \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") "
Jul 01 21:10:54.150224 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150045    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcpq8\" (UniqueName: \"kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8\") pod \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") "
Jul 01 21:10:54.150224 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150065    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config\") pod \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\" (UID: \"28adbd4b-9edd-49bf-aede-1ad7f0e0366f\") "
Jul 01 21:10:54.150371 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150345    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28adbd4b-9edd-49bf-aede-1ad7f0e0366f" (UID: "28adbd4b-9edd-49bf-aede-1ad7f0e0366f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:54.150516 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150402    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28adbd4b-9edd-49bf-aede-1ad7f0e0366f" (UID: "28adbd4b-9edd-49bf-aede-1ad7f0e0366f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:10:54.150632 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.150553    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28adbd4b-9edd-49bf-aede-1ad7f0e0366f" (UID: "28adbd4b-9edd-49bf-aede-1ad7f0e0366f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:10:54.152269 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.152250    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28adbd4b-9edd-49bf-aede-1ad7f0e0366f" (UID: "28adbd4b-9edd-49bf-aede-1ad7f0e0366f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:10:54.152374 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.152354    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8" (OuterVolumeSpecName: "kube-api-access-wcpq8") pod "28adbd4b-9edd-49bf-aede-1ad7f0e0366f" (UID: "28adbd4b-9edd-49bf-aede-1ad7f0e0366f"). InnerVolumeSpecName "kube-api-access-wcpq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:10:54.250883 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.250854    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kserve-provision-location\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:10:54.250883 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.250880    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-proxy-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:10:54.251055 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.250893    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-cabundle-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:10:54.251055 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.250905    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpq8\" (UniqueName: \"kubernetes.io/projected/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-api-access-wcpq8\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:10:54.251055 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.250916    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28adbd4b-9edd-49bf-aede-1ad7f0e0366f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:10:54.378624 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.378593    2574 generic.go:356] "Generic (PLEG): container finished" podID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerID="73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992" exitCode=0
Jul 01 21:10:54.378803 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.378631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerDied","Data":"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"}
Jul 01 21:10:54.378803 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.378649    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"
Jul 01 21:10:54.378803 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.378679    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc" event={"ID":"28adbd4b-9edd-49bf-aede-1ad7f0e0366f","Type":"ContainerDied","Data":"8ee1ffbfef04388553627f650166f749df710786ba64f24f0250b9416350f26e"}
Jul 01 21:10:54.378803 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.378704    2574 scope.go:117] "RemoveContainer" containerID="e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"
Jul 01 21:10:54.392832 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.392810    2574 scope.go:117] "RemoveContainer" containerID="73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"
Jul 01 21:10:54.400322 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.400306    2574 scope.go:117] "RemoveContainer" containerID="453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"
Jul 01 21:10:54.405386 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.405369    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:10:54.407977 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.407921    2574 scope.go:117] "RemoveContainer" containerID="e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"
Jul 01 21:10:54.408290 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:10:54.408267    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15\": container with ID starting with e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15 not found: ID does not exist" containerID="e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"
Jul 01 21:10:54.408348 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408299    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15"} err="failed to get container status \"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15\": rpc error: code = NotFound desc = could not find container \"e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15\": container with ID starting with e9d35f6cc7645fbfe073e631d829637796d9e5d4f780254c8b110ab7aff00a15 not found: ID does not exist"
Jul 01 21:10:54.408348 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408323    2574 scope.go:117] "RemoveContainer" containerID="73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"
Jul 01 21:10:54.408625 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:10:54.408606    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992\": container with ID starting with 73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992 not found: ID does not exist" containerID="73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"
Jul 01 21:10:54.408673 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408630    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992"} err="failed to get container status \"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992\": rpc error: code = NotFound desc = could not find container \"73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992\": container with ID starting with 73c4f3f54205cbbd252c378ad6e2277e9ab4f67402b860b6389316be342b4992 not found: ID does not exist"
Jul 01 21:10:54.408673 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408645    2574 scope.go:117] "RemoveContainer" containerID="453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"
Jul 01 21:10:54.408941 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:10:54.408922    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25\": container with ID starting with 453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25 not found: ID does not exist" containerID="453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"
Jul 01 21:10:54.408986 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408948    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25"} err="failed to get container status \"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25\": rpc error: code = NotFound desc = could not find container \"453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25\": container with ID starting with 453e2b4312ff379e16e2857acc476a0401bea947fcbce8a15a84994847d12d25 not found: ID does not exist"
Jul 01 21:10:54.409024 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:54.408994    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc"]
Jul 01 21:10:55.461450 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461421    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ncvhv/must-gather-mw4tg"]
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461777    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container"
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461789    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container"
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461805    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="storage-initializer"
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461811    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="storage-initializer"
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461819    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kube-rbac-proxy"
Jul 01 21:10:55.461847 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461824    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kube-rbac-proxy"
Jul 01 21:10:55.462035 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461880    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kserve-container"
Jul 01 21:10:55.462035 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.461888    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" containerName="kube-rbac-proxy"
Jul 01 21:10:55.466149 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.466126    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.468725 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.468689    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ncvhv\"/\"default-dockercfg-hmx5q\""
Jul 01 21:10:55.468860 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.468742    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ncvhv\"/\"openshift-service-ca.crt\""
Jul 01 21:10:55.469209 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.469194    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ncvhv\"/\"kube-root-ca.crt\""
Jul 01 21:10:55.475746 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.475724    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncvhv/must-gather-mw4tg"]
Jul 01 21:10:55.482909 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.482889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:10:55.548177 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.548154    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28adbd4b-9edd-49bf-aede-1ad7f0e0366f" path="/var/lib/kubelet/pods/28adbd4b-9edd-49bf-aede-1ad7f0e0366f/volumes"
Jul 01 21:10:55.558494 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.558474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.558580 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.558527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwhs\" (UniqueName: \"kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.581142 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.581120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:10:55.659848 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.659821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.659956 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.659897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwhs\" (UniqueName: \"kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.660171 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.660152    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.668808 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.668790    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwhs\" (UniqueName: \"kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs\") pod \"must-gather-mw4tg\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") " pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.787388 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.787309    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:10:55.913632 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.913602    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncvhv/must-gather-mw4tg"]
Jul 01 21:10:55.917300 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:10:55.917274    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb03e4b3_de50_4b0b_9515_218134946ef3.slice/crio-ee8498790c0894517ba1252e57862a57d67fdba07fd3fac74bd282016c3a1db5 WatchSource:0}: Error finding container ee8498790c0894517ba1252e57862a57d67fdba07fd3fac74bd282016c3a1db5: Status 404 returned error can't find the container with id ee8498790c0894517ba1252e57862a57d67fdba07fd3fac74bd282016c3a1db5
Jul 01 21:10:55.918978 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:55.918964    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 21:10:56.389189 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:10:56.389156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" event={"ID":"cb03e4b3-de50-4b0b-9515-218134946ef3","Type":"ContainerStarted","Data":"ee8498790c0894517ba1252e57862a57d67fdba07fd3fac74bd282016c3a1db5"}
Jul 01 21:11:02.418367 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:02.418330    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" event={"ID":"cb03e4b3-de50-4b0b-9515-218134946ef3","Type":"ContainerStarted","Data":"9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea"}
Jul 01 21:11:02.418367 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:02.418375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" event={"ID":"cb03e4b3-de50-4b0b-9515-218134946ef3","Type":"ContainerStarted","Data":"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"}
Jul 01 21:11:02.433928 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:02.433881    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" podStartSLOduration=1.50111562 podStartE2EDuration="7.433866285s" totalImagesPullingTime="5.932750665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:02.433476471 +0000 UTC m=+1453.348754510" watchObservedRunningTime="2026-07-01 21:11:02.433866285 +0000 UTC m=+1453.349144323"
Jul 01 21:11:23.507350 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:23.507270    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerID="5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1" exitCode=0
Jul 01 21:11:23.507846 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:23.507344    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" event={"ID":"cb03e4b3-de50-4b0b-9515-218134946ef3","Type":"ContainerDied","Data":"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"}
Jul 01 21:11:23.507846 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:23.507743    2574 scope.go:117] "RemoveContainer" containerID="5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"
Jul 01 21:11:24.236017 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.235990    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ncvhv_must-gather-mw4tg_cb03e4b3-de50-4b0b-9515-218134946ef3/gather/0.log"
Jul 01 21:11:24.757729 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.757683    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ld7vj/must-gather-j4fhv"]
Jul 01 21:11:24.760284 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.760268    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.763142 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.763122    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ld7vj\"/\"openshift-service-ca.crt\""
Jul 01 21:11:24.763260 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.763122    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ld7vj\"/\"kube-root-ca.crt\""
Jul 01 21:11:24.763260 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.763183    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ld7vj\"/\"default-dockercfg-k5zts\""
Jul 01 21:11:24.773231 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.770653    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ld7vj/must-gather-j4fhv"]
Jul 01 21:11:24.784432 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.784410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46gs\" (UniqueName: \"kubernetes.io/projected/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-kube-api-access-s46gs\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.784543 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.784447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-must-gather-output\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.884920 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.884896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46gs\" (UniqueName: \"kubernetes.io/projected/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-kube-api-access-s46gs\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.885034 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.884930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-must-gather-output\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.885300 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.885281    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-must-gather-output\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:24.893191 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:24.893169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46gs\" (UniqueName: \"kubernetes.io/projected/d4bc35c1-6304-4e21-bf9f-0039dd267b9f-kube-api-access-s46gs\") pod \"must-gather-j4fhv\" (UID: \"d4bc35c1-6304-4e21-bf9f-0039dd267b9f\") " pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:25.075570 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:25.075510    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ld7vj/must-gather-j4fhv"
Jul 01 21:11:25.203964 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:25.203941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ld7vj/must-gather-j4fhv"]
Jul 01 21:11:25.206364 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:11:25.206332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4bc35c1_6304_4e21_bf9f_0039dd267b9f.slice/crio-af6b2c682290d3e8cb95f468b24a1dfe6f9386607d64eb748286086c2839903a WatchSource:0}: Error finding container af6b2c682290d3e8cb95f468b24a1dfe6f9386607d64eb748286086c2839903a: Status 404 returned error can't find the container with id af6b2c682290d3e8cb95f468b24a1dfe6f9386607d64eb748286086c2839903a
Jul 01 21:11:25.288233 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:25.288209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:11:25.348118 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:25.348034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:11:25.515256 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:25.515222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ld7vj/must-gather-j4fhv" event={"ID":"d4bc35c1-6304-4e21-bf9f-0039dd267b9f","Type":"ContainerStarted","Data":"af6b2c682290d3e8cb95f468b24a1dfe6f9386607d64eb748286086c2839903a"}
Jul 01 21:11:26.522594 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:26.521818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ld7vj/must-gather-j4fhv" event={"ID":"d4bc35c1-6304-4e21-bf9f-0039dd267b9f","Type":"ContainerStarted","Data":"3a0cfeffa21dc1c9a94f80818a5f6bbfc1df394e9bb1d5554ac18562f5a2dd4b"}
Jul 01 21:11:26.522594 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:26.521870    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ld7vj/must-gather-j4fhv" event={"ID":"d4bc35c1-6304-4e21-bf9f-0039dd267b9f","Type":"ContainerStarted","Data":"a3e2eb4d4b2b46c3c97e93c05a1ec57211912f6641c297f804520ac16c461479"}
Jul 01 21:11:26.541375 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:26.541314    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ld7vj/must-gather-j4fhv" podStartSLOduration=1.868655309 podStartE2EDuration="2.541293443s" totalImagesPullingTime="672.638134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:26.539579725 +0000 UTC m=+1477.454857764" watchObservedRunningTime="2026-07-01 21:11:26.541293443 +0000 UTC m=+1477.456571483"
Jul 01 21:11:27.316491 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:27.316458    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-77s9m_68e6e793-e805-4c65-8b94-5d540f3bf401/global-pull-secret-syncer/0.log"
Jul 01 21:11:27.502262 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:27.502232    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log"
Jul 01 21:11:27.586528 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:27.586445    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-188.ec2.internal_a5e0ed2ae21cf9ca9826a478a311bf2a/haproxy/0.log"
Jul 01 21:11:29.606243 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:29.606202    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ncvhv/must-gather-mw4tg"]
Jul 01 21:11:29.606761 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:29.606440    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-ncvhv/must-gather-mw4tg" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="copy" containerID="cri-o://9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea" gracePeriod=2
Jul 01 21:11:29.613386 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:29.612692    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ncvhv/must-gather-mw4tg"]
Jul 01 21:11:29.959681 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:29.958643    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ncvhv_must-gather-mw4tg_cb03e4b3-de50-4b0b-9515-218134946ef3/copy/0.log"
Jul 01 21:11:29.961402 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:29.961165    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:11:29.963919 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:11:29.963889    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mw4tg\" is forbidden: User \"system:node:ip-10-0-135-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ncvhv\": no relationship found between node 'ip-10-0-135-188.ec2.internal' and this object" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:11:30.033826 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.033783    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlwhs\" (UniqueName: \"kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs\") pod \"cb03e4b3-de50-4b0b-9515-218134946ef3\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") "
Jul 01 21:11:30.034091 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.034068    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output\") pod \"cb03e4b3-de50-4b0b-9515-218134946ef3\" (UID: \"cb03e4b3-de50-4b0b-9515-218134946ef3\") "
Jul 01 21:11:30.036843 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.036818    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cb03e4b3-de50-4b0b-9515-218134946ef3" (UID: "cb03e4b3-de50-4b0b-9515-218134946ef3"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:30.045516 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.045491    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs" (OuterVolumeSpecName: "kube-api-access-hlwhs") pod "cb03e4b3-de50-4b0b-9515-218134946ef3" (UID: "cb03e4b3-de50-4b0b-9515-218134946ef3"). InnerVolumeSpecName "kube-api-access-hlwhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:11:30.135697 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.135666    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb03e4b3-de50-4b0b-9515-218134946ef3-must-gather-output\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:11:30.135972 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.135949    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwhs\" (UniqueName: \"kubernetes.io/projected/cb03e4b3-de50-4b0b-9515-218134946ef3-kube-api-access-hlwhs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 01 21:11:30.553451 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.553363    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ncvhv_must-gather-mw4tg_cb03e4b3-de50-4b0b-9515-218134946ef3/copy/0.log"
Jul 01 21:11:30.556867 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.555803    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerID="9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea" exitCode=143
Jul 01 21:11:30.556867 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.555958    2574 scope.go:117] "RemoveContainer" containerID="9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea"
Jul 01 21:11:30.556867 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.556125    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:11:30.571324 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:11:30.571293    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mw4tg\" is forbidden: User \"system:node:ip-10-0-135-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ncvhv\": no relationship found between node 'ip-10-0-135-188.ec2.internal' and this object" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:11:30.583944 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:11:30.583912    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mw4tg\" is forbidden: User \"system:node:ip-10-0-135-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ncvhv\": no relationship found between node 'ip-10-0-135-188.ec2.internal' and this object" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" pod="openshift-must-gather-ncvhv/must-gather-mw4tg"
Jul 01 21:11:30.586843 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.586818    2574 scope.go:117] "RemoveContainer" containerID="5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"
Jul 01 21:11:30.612477 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.612442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:11:30.622280 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.622131    2574 scope.go:117] "RemoveContainer" containerID="9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea"
Jul 01 21:11:30.622674 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:11:30.622635    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea\": container with ID starting with 9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea not found: ID does not exist" containerID="9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea"
Jul 01 21:11:30.622877 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.622853    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea"} err="failed to get container status \"9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea\": rpc error: code = NotFound desc = could not find container \"9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea\": container with ID starting with 9313c185c3b847750851a03cd16cf0f011ab4357696bc7edbe1f634fbf3eb7ea not found: ID does not exist"
Jul 01 21:11:30.623027 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.623016    2574 scope.go:117] "RemoveContainer" containerID="5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"
Jul 01 21:11:30.623563 ip-10-0-135-188 kubenswrapper[2574]: E0701 21:11:30.623542    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1\": container with ID starting with 5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1 not found: ID does not exist" containerID="5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"
Jul 01 21:11:30.623771 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.623750    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1"} err="failed to get container status \"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1\": rpc error: code = NotFound desc = could not find container \"5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1\": container with ID starting with 5c5cc0b3cc6000e46667c8f8e5ea5ede3af79c82bcb3d8d563f57c06aae547a1 not found: ID does not exist"
Jul 01 21:11:30.678168 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:30.678124    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2585q_fffcba5b-286c-4bca-9cc5-173f6279f09f/konnectivity-agent/0.log" limit=1024
Jul 01 21:11:31.551654 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:31.551617    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" path="/var/lib/kubelet/pods/cb03e4b3-de50-4b0b-9515-218134946ef3/volumes"
Jul 01 21:11:31.970698 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:31.970655    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-pwgkr_5deae4be-fe60-4f24-8415-da74dace7ff4/kube-state-metrics/0.log"
Jul 01 21:11:32.001045 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.001017    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-pwgkr_5deae4be-fe60-4f24-8415-da74dace7ff4/kube-rbac-proxy-main/0.log"
Jul 01 21:11:32.041319 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.041284    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-pwgkr_5deae4be-fe60-4f24-8415-da74dace7ff4/kube-rbac-proxy-self/0.log"
Jul 01 21:11:32.363032 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.362997    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9vrh_c73f447e-3bdc-4af4-9cee-32591ebd060e/node-exporter/0.log"
Jul 01 21:11:32.387566 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.387543    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9vrh_c73f447e-3bdc-4af4-9cee-32591ebd060e/kube-rbac-proxy/0.log"
Jul 01 21:11:32.414817 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.414787    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9vrh_c73f447e-3bdc-4af4-9cee-32591ebd060e/init-textfile/0.log"
Jul 01 21:11:32.448619 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.448586    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2kw5z_8f6a7cde-81af-40d7-954f-cbdebb51075b/kube-rbac-proxy-main/0.log"
Jul 01 21:11:32.475190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.475165    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2kw5z_8f6a7cde-81af-40d7-954f-cbdebb51075b/kube-rbac-proxy-self/0.log"
Jul 01 21:11:32.499311 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.499266    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2kw5z_8f6a7cde-81af-40d7-954f-cbdebb51075b/openshift-state-metrics/0.log"
Jul 01 21:11:32.537209 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.537138    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/prometheus/0.log"
Jul 01 21:11:32.562521 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.562485    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/config-reloader/0.log"
Jul 01 21:11:32.599144 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.599114    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/thanos-sidecar/0.log"
Jul 01 21:11:32.627126 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.627090    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/kube-rbac-proxy-web/0.log"
Jul 01 21:11:32.652672 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.652647    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/kube-rbac-proxy/0.log"
Jul 01 21:11:32.676201 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.676163    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/kube-rbac-proxy-thanos/0.log"
Jul 01 21:11:32.700451 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.700422    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce07ba80-c7de-47d5-895e-2f0a1b6b2cdb/init-config-reloader/0.log"
Jul 01 21:11:32.740955 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.740921    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-jlxhh_7ee3ab90-3ee4-4244-84f7-f34e66edf21f/prometheus-operator/0.log"
Jul 01 21:11:32.763418 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.763395    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-jlxhh_7ee3ab90-3ee4-4244-84f7-f34e66edf21f/kube-rbac-proxy/0.log"
Jul 01 21:11:32.927020 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.926933    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/thanos-query/0.log"
Jul 01 21:11:32.958933 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.958901    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/kube-rbac-proxy-web/0.log"
Jul 01 21:11:32.987790 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:32.987760    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/kube-rbac-proxy/0.log"
Jul 01 21:11:33.022132 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:33.022101    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/prom-label-proxy/0.log"
Jul 01 21:11:33.094524 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:33.094437    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/kube-rbac-proxy-rules/0.log"
Jul 01 21:11:33.153003 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:33.152972    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cd9f75-tb7n8_26c6cad7-b26a-40ae-b673-25d3a593a741/kube-rbac-proxy-metrics/0.log"
Jul 01 21:11:35.400364 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.400331    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"]
Jul 01 21:11:35.402054 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402029    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="gather"
Jul 01 21:11:35.402054 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402056    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="gather"
Jul 01 21:11:35.402190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402080    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="copy"
Jul 01 21:11:35.402190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402089    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="copy"
Jul 01 21:11:35.402190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402152    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="gather"
Jul 01 21:11:35.402190 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.402169    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb03e4b3-de50-4b0b-9515-218134946ef3" containerName="copy"
Jul 01 21:11:35.406363 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.406337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.412980 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.412951    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"]
Jul 01 21:11:35.508577 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.508542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-sys\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.508930 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.508901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-lib-modules\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.509125 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.509108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxdl\" (UniqueName: \"kubernetes.io/projected/934a8b8f-2904-42af-bb75-a7f651c02be8-kube-api-access-xpxdl\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.509260 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.509245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-proc\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.509362 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.509350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-podres\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610040 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-lib-modules\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610040 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxdl\" (UniqueName: \"kubernetes.io/projected/934a8b8f-2904-42af-bb75-a7f651c02be8-kube-api-access-xpxdl\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-proc\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-podres\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610255    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-proc\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610282 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610191    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-lib-modules\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610467 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-sys\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610467 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610386    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-podres\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.610467 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.610412    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/934a8b8f-2904-42af-bb75-a7f651c02be8-sys\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.619445 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.619416    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxdl\" (UniqueName: \"kubernetes.io/projected/934a8b8f-2904-42af-bb75-a7f651c02be8-kube-api-access-xpxdl\") pod \"perf-node-gather-daemonset-89rfc\" (UID: \"934a8b8f-2904-42af-bb75-a7f651c02be8\") " pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.725396 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.725362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:35.889688 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:35.889491    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"]
Jul 01 21:11:35.892812 ip-10-0-135-188 kubenswrapper[2574]: W0701 21:11:35.892782    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod934a8b8f_2904_42af_bb75_a7f651c02be8.slice/crio-a64609e66499c895327eeac2278b8b02f2160355a920dd41ab722bfdc5b00e5f WatchSource:0}: Error finding container a64609e66499c895327eeac2278b8b02f2160355a920dd41ab722bfdc5b00e5f: Status 404 returned error can't find the container with id a64609e66499c895327eeac2278b8b02f2160355a920dd41ab722bfdc5b00e5f
Jul 01 21:11:36.590525 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.590479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc" event={"ID":"934a8b8f-2904-42af-bb75-a7f651c02be8","Type":"ContainerStarted","Data":"e0ac045de920ce74e883e2306c8a85c246e43f800cdf63f56667358732aafe1d"}
Jul 01 21:11:36.591007 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.590554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc" event={"ID":"934a8b8f-2904-42af-bb75-a7f651c02be8","Type":"ContainerStarted","Data":"a64609e66499c895327eeac2278b8b02f2160355a920dd41ab722bfdc5b00e5f"}
Jul 01 21:11:36.591358 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.591335    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:36.610214 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.609978    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc" podStartSLOduration=1.609960142 podStartE2EDuration="1.609960142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:36.607410136 +0000 UTC m=+1487.522688178" watchObservedRunningTime="2026-07-01 21:11:36.609960142 +0000 UTC m=+1487.525238181"
Jul 01 21:11:36.763168 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.763132    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnjkr_8ba42c95-d563-4e39-b778-6a76b43761e3/dns/0.log"
Jul 01 21:11:36.785527 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.785499    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnjkr_8ba42c95-d563-4e39-b778-6a76b43761e3/kube-rbac-proxy/0.log"
Jul 01 21:11:36.912503 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:36.912430    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5h5gr_e868c3e6-e4a7-4289-9f1a-49d57e56a504/dns-node-resolver/0.log"
Jul 01 21:11:37.484638 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:37.484616    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kgp25_0a2f09f4-40c7-410f-a2f4-6fde0343f956/node-ca/0.log"
Jul 01 21:11:38.801439 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:38.801411    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6jx6_464150bb-e3fb-42f5-881e-a3e4f64b08f4/serve-healthcheck-canary/0.log"
Jul 01 21:11:39.555090 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:39.555067    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ktjs7_7ed6e45f-4729-4d51-9e0f-1b2c8013cd56/kube-rbac-proxy/0.log"
Jul 01 21:11:39.586983 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:39.586959    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ktjs7_7ed6e45f-4729-4d51-9e0f-1b2c8013cd56/exporter/0.log"
Jul 01 21:11:39.615382 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:39.615348    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ktjs7_7ed6e45f-4729-4d51-9e0f-1b2c8013cd56/extractor/0.log"
Jul 01 21:11:41.829034 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:41.829006    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-cwvfk_b06fdbbf-8b20-44bc-9924-b03cf338c82e/server/0.log"
Jul 01 21:11:41.940210 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:41.940167    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bn8nq_c523d6e4-de16-45d8-a73e-58df48bb156d/s3-init/0.log"
Jul 01 21:11:41.957476 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:41.957451    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bn8nq_c523d6e4-de16-45d8-a73e-58df48bb156d/download-hf-model/0.log"
Jul 01 21:11:41.985726 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:41.985682    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-krbb5_eab2b0a9-e895-4e31-befe-e007b8ffa0a6/seaweedfs/0.log"
Jul 01 21:11:43.610991 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:43.610960    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ld7vj/perf-node-gather-daemonset-89rfc"
Jul 01 21:11:47.630830 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.630793    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/kube-multus-additional-cni-plugins/0.log"
Jul 01 21:11:47.655787 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.655756    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/egress-router-binary-copy/0.log"
Jul 01 21:11:47.681080 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.681055    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/cni-plugins/0.log"
Jul 01 21:11:47.705112 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.705086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/bond-cni-plugin/0.log"
Jul 01 21:11:47.728272 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.728239    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/routeoverride-cni/0.log"
Jul 01 21:11:47.750483 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.750454    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/whereabouts-cni-bincopy/0.log"
Jul 01 21:11:47.773299 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:47.773270    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dzdqd_be17cc30-0448-48a7-aea8-cc056b04e27e/whereabouts-cni/0.log"
Jul 01 21:11:48.018246 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:48.018220    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fs6qt_6fac2d09-f4b9-46b4-85a1-cf5f71bd34dd/kube-multus/0.log"
Jul 01 21:11:48.047369 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:48.047340    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9jvqx_4c08e116-b49a-4f87-a27c-2e4a5b0cfece/network-metrics-daemon/0.log"
Jul 01 21:11:48.071645 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:48.071625    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9jvqx_4c08e116-b49a-4f87-a27c-2e4a5b0cfece/kube-rbac-proxy/0.log"
Jul 01 21:11:49.189864 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.189834    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-controller/0.log"
Jul 01 21:11:49.210170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.210150    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:11:49.210170 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.210166    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/1.log"
Jul 01 21:11:49.229882 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.229862    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/kube-rbac-proxy-node/0.log"
Jul 01 21:11:49.260948 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.260931    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 21:11:49.292768 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.292747    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/northd/0.log"
Jul 01 21:11:49.322625 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.322585    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/nbdb/0.log"
Jul 01 21:11:49.354915 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.354889    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/sbdb/0.log"
Jul 01 21:11:49.408122 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.408103    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovnkube-controller/0.log"
Jul 01 21:11:49.612330 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.612263    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:11:49.612330 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:49.612302    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7jjmd_a6412b14-0c6f-48e8-b6c8-6011fe7228a7/ovn-acl-logging/0.log"
Jul 01 21:11:50.879289 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:50.879265    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8csxj_2eb44aee-dfbc-45b7-9ee4-45ee9acbd117/network-check-target-container/0.log"
Jul 01 21:11:51.940551 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:51.940518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dm65v_e42e0e9f-1686-4da6-937e-af4809a93f27/iptables-alerter/0.log"
Jul 01 21:11:52.660352 ip-10-0-135-188 kubenswrapper[2574]: I0701 21:11:52.660329    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-t5svc_fc1e123a-a344-4a76-88d2-3c0db03dc084/tuned/0.log"