Jul 28 16:02:33.325631 ip-10-0-142-59 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 16:02:33.723778 ip-10-0-142-59 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:02:33.723778 ip-10-0-142-59 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 16:02:33.723778 ip-10-0-142-59 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:02:33.723778 ip-10-0-142-59 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 16:02:33.723778 ip-10-0-142-59 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:02:33.724624 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.723889    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 16:02:33.728911 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728888    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.728911 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728908    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.728911 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728914    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.728911 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728918    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728923    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728928    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728933    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728936    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728942    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728946    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728950    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728953    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728957    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728960    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728964    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728969    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728973    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728976    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728980    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728985    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728989    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728993    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.728997    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.729172 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729001    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729005    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729009    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729013    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729017    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729022    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729026    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729029    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729034    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729037    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729041    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729045    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729049    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729054    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729061    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729067    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729071    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729075    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729079    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729084    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.729936 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729089    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729093    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729097    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729102    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729106    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729110    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729114    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729119    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729123    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729127    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729131    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729136    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729140    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729145    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729150    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729154    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729159    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729163    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729167    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729171    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.730477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729175    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729179    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729188    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729194    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729199    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729205    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729213    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729222    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729229    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729235    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729240    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729245    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729249    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729253    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729258    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729262    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729266    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729270    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.730994 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729275    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729279    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729283    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729286    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729291    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729296    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729300    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729305    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729310    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729314    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729318    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729323    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729328    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729333    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729337    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729341    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.729345    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730360    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730372    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730376    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.731454 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730379    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730383    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730386    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730388    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730391    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730394    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730397    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730400    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730403    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730405    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730407    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730410    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730413    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730415    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730418    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730420    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730424    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730427    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730429    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730432    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.731962 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730434    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730436    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730439    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730442    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730444    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730446    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730449    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730452    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730455    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730457    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730462    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730464    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730467    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730470    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730472    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730475    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730477    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730480    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730483    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730485    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.732449 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730488    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730491    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730493    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730496    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730499    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730502    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730504    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730507    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730509    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730512    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730514    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730517    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730520    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730524    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730528    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730532    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730534    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730537    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730540    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.732956 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730543    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730545    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730548    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730551    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730555    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730557    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730560    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730562    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730565    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730568    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730570    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730573    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730576    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730578    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730581    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730583    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730587    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730590    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730593    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730596    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.733444 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730599    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730602    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730604    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730607    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730610    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730612    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730617    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730622    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730625    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730627    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730630    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730633    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730635    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730638    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730641    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730644    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.730647    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731052    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731058    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.733942 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731061    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731064    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731066    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731069    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731072    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731074    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731077    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731079    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731082    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731085    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731088    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731090    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731093    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731098    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731101    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731104    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731108    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731111    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731114    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.734416 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731117    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731120    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731123    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731125    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731128    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731131    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731134    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731136    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731139    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731144    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731147    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731150    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731153    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731155    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731158    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731160    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731163    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731166    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731168    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731171    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.734883 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731174    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731176    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731179    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731182    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731184    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731187    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731189    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731192    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731194    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731197    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731199    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731202    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731206    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731209    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731211    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731214    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731216    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731220    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731222    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731225    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.735367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731227    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731231    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731234    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731237    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731240    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731242    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731245    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731247    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731250    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731252    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731256    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731259    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731262    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731265    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731269    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731272    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731275    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731278    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.735875 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731281    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731284    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731287    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731289    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731292    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731294    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731298    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731300    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731303    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731306    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731308    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731311    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731313    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731316    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731318    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731322    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731325    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731327    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731330    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.731333    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731401    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 16:02:33.736345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731409    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731416    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731420    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731425    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731428    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731432    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731437    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731440    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731443    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731447    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731450    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731453    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731456    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731460    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731463    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731466    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731469    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731474    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731477    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731480    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731483    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731487    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731491    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 16:02:33.736887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731494    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731498    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731502    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731506    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731509    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731512    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731515    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731518    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731523    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731526    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731529    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731531    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731534    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731537    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731542    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731545    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731548    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731551    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731554    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731558    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731561    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731564    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731567    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731570    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731573    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 16:02:33.737544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731576    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731579    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731582    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731586    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731589    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731593    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731596    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731599    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731603    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731606    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731609    2579 flags.go:64] FLAG: --help="false"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731611    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-59.ec2.internal"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731614    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731617    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731620    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731624    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731628    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731631    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731634    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731637    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731640    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731643    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731646    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731648    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 16:02:33.738178 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731651    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731654    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731657    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731660    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731663    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731666    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731669    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731672    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731677    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731681    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731684    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731686    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731689    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731692    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731695    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731698    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731703    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731706    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731710    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731713    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731715    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731718    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731721    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731725    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731728    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 16:02:33.738787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731731    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731738    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731741    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731745    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731748    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731751    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731757    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731760    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731763    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731766    2579 flags.go:64] FLAG: --port="10250"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731769    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731772    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddddbcac221ae1d7"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731776    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731779    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731781    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731784    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731788    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731791    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731794    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731797    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731801    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731804    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731807    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731809    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731812    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 16:02:33.739408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731815    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731818    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731821    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731824    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731827    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731829    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731833    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731835    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731839    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731842    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731859    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731863    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731870    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731874    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731877    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731882    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731885    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731889    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731893    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731896    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731899    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731902    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731905    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731908    2579 flags.go:64] FLAG: --v="2"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731912    2579 flags.go:64] FLAG: --version="false"
Jul 28 16:02:33.740022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731916    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731920    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.731923    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732045    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732048    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732051    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732054    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732057    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732060    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732063    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732066    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732069    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732072    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732076    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732079    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732082    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732086    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732088    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732091    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.740669 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732094    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732097    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732099    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732102    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732105    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732108    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732111    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732113    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732116    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732118    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732121    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732123    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732126    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732128    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732131    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732134    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732136    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732139    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732141    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.741322 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732144    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732147    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732149    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732151    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732154    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732156    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732159    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732162    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732164    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732166    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732169    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732172    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732174    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732178    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732181    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732184    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732186    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732189    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732193    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732195    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.742083 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732198    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732201    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732203    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732206    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732208    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732210    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732214    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732218    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732221    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732225    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732228    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732231    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732233    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732236    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732239    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732241    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732244    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732246    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732248    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732251    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.742717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732254    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732256    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732258    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732261    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732264    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732270    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732273    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732275    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732278    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732280    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732284    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732286    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732289    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732291    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732294    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732296    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732299    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732301    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732304    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732307    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.743275 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732309    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732312    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732314    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.732317    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.733043    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.740089    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.740109    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.740122    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.740128    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740200    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740204    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740208    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740211    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740214    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740217    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.743767 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740220    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740222    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740225    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740228    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740231    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740233    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740236    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740239    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740242    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740244    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740248    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740252    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740255    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740257    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740260    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740263    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740266    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740268    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740271    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740273    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.744151 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740276    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740278    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740281    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740289    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740292    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740295    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740297    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740299    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740302    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740305    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740307    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740310    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740313    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740315    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740317    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740320    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740322    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740325    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740327    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740330    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.744672 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740333    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740335    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740338    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740340    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740343    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740345    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740348    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740350    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740352    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740355    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740357    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740360    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740363    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740365    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740368    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740370    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740379    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740382    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740386    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740389    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.745211 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740392    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740394    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740397    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740400    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740403    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740405    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740408    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740410    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740413    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740415    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740418    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740420    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740423    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740425    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740428    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740430    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740433    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740435    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740438    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.745713 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740446    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740448    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740452    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740455    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740458    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740461    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740464    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740467    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740469    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740472    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740480    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740482    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740485    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740487    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.740492    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:02:33.746197 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740915    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740928    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740933    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740941    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740947    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740958    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740963    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740968    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740973    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740977    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740981    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740986    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740990    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740994    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.740998    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741002    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741007    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741011    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741019    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:02:33.746609 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741024    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741028    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741032    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741036    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741040    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741044    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741048    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741053    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741057    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741091    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741096    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741105    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741109    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741113    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741118    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741122    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741126    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741131    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741135    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741140    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:02:33.747094 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741146    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741151    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741155    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741160    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741169    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741173    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741177    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741181    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741186    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741190    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741195    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741199    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741203    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741208    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741212    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741216    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741225    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741229    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741233    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741240    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:02:33.747570 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741243    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741251    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741285    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741293    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741298    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741302    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741308    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741312    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741316    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741328    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741333    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741338    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741343    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741347    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741351    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741356    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741360    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741364    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741368    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741372    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:02:33.748072 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741377    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741386    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741390    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741395    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741399    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741403    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741410    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741415    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741421    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741426    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741431    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741437    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741441    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741450    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741454    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741487    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741491    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741496    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741500    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:02:33.748567 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:33.741504    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.741512    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.742271    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.744427    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.745700    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.745797    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.745832    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:02:33.749047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.748225    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:02:33.749396 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.749379    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:02:33.763755 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.763733    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 16:02:33.765875 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.765860    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 16:02:33.766621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.766600    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 16:02:33.768876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.768837    2579 fs.go:135] Filesystem UUIDs: map[2db194ec-a79d-43a5-a89c-d1ee74c261be:/dev/nvme0n1p3 6f25a7e8-f155-4d3c-a0a5-200919056576:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 16:02:33.768966 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.768876    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 16:02:33.773769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.773749    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:02:33.775522 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.775411    2579 manager.go:216] Machine: {Timestamp:2026-07-28 16:02:33.774160223 +0000 UTC m=+0.325758207 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098709 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec294068c8d0522838dbcb2afafe43af SystemUUID:ec294068-c8d0-5228-38db-cb2afafe43af BootID:80964e89-0a7c-4990-ac10-a8c75c4c1c71 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:f8:f3:b2:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:f8:f3:b2:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:7c:0f:f3:b3:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 16:02:33.775522 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.775521    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 16:02:33.775648 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.775633    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 16:02:33.779191 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.779167    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 16:02:33.779326 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.779194    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-59.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 16:02:33.780365 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.780350    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 16:02:33.780399 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.780374    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 16:02:33.780399 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.780381    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:02:33.780975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.780964    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:02:33.781696 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.781674    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 16:02:33.783539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783529    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:02:33.783652 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783642    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 16:02:33.783686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783671    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 16:02:33.783716 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783701    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 16:02:33.783716 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783715    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 16:02:33.783772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783768    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 16:02:33.783802 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.783783    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 16:02:33.784397 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.784374    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:02:33.784489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.784401    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:02:33.788488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.788469    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 16:02:33.790219 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.790206    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 16:02:33.790286 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.790236    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 16:02:33.791729 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791715    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 16:02:33.791774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791741    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 16:02:33.791774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791753    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 16:02:33.791774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791760    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 16:02:33.791774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791768    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 16:02:33.791774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791774    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791782    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791787    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791794    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791799    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791808    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 16:02:33.791932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.791813    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 16:02:33.792582 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.792572    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 16:02:33.792582 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.792581    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 16:02:33.793830 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.793803    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 16:02:33.793908 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.793811    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 16:02:33.794037 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.794016    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-59.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 16:02:33.795919 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.795907    2579 server.go:1268] "Started kubelet"
Jul 28 16:02:33.796054 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.796016    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 16:02:33.796211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.796154    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 16:02:33.796275 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.796236    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 16:02:33.796781 ip-10-0-142-59 systemd[1]: Started Kubernetes Kubelet.
Jul 28 16:02:33.797809 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.797740    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 16:02:33.798920 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.798908    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 16:02:33.801787 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.801756    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 16:02:33.802025 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.800590    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-59.ec2.internal.18c67f8ed8a6ec1d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-59.ec2.internal,UID:ip-10-0-142-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-59.ec2.internal,},FirstTimestamp:2026-07-28 16:02:33.795882013 +0000 UTC m=+0.347479997,LastTimestamp:2026-07-28 16:02:33.795882013 +0000 UTC m=+0.347479997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-59.ec2.internal,}"
Jul 28 16:02:33.803592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.803572    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 16:02:33.803592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.803588    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 16:02:33.803724 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.803666    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:33.804192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804174    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 16:02:33.804192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804176    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 16:02:33.804334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804203    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 16:02:33.804334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804295    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 16:02:33.804334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804304    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 16:02:33.804517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804503    2579 factory.go:55] Registering systemd factory
Jul 28 16:02:33.804570 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804522    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 16:02:33.804769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804755    2579 factory.go:153] Registering CRI-O factory
Jul 28 16:02:33.804830 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804774    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 16:02:33.804830 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804823    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 16:02:33.804916 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804842    2579 factory.go:103] Registering Raw factory
Jul 28 16:02:33.804916 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.804884    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 16:02:33.805365 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.805351    2579 manager.go:349] Starting recovery of all containers
Jul 28 16:02:33.807090 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.807054    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ncdw"
Jul 28 16:02:33.814121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.814096    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ncdw"
Jul 28 16:02:33.814500 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.814472    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 16:02:33.814638 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.814616    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 16:02:33.816216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.816194    2579 manager.go:354] Recovery completed
Jul 28 16:02:33.821973 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.821961    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:33.824013 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.823935    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:33.824013 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.823966    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:33.824013 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.823977    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:33.824539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.824521    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 16:02:33.824539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.824540    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 16:02:33.824640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.824561    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:02:33.824640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.824530    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:33.825257 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.825240    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:33.825340 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.825267    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:33.825340 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.825277    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:33.826195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.826181    2579 policy_none.go:49] "None policy: Start"
Jul 28 16:02:33.826195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.826197    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 16:02:33.826334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.826208    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 16:02:33.827411 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.827395    2579 policy_none.go:47] "Start"
Jul 28 16:02:33.857401 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857386    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 16:02:33.857489 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.857417    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 16:02:33.857489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857427    2579 server.go:87] "Starting device plugin registration server"
Jul 28 16:02:33.857678 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857667    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 16:02:33.857714 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857680    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 16:02:33.857783 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857767    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 16:02:33.857902 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857885    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 16:02:33.857902 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.857904    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 16:02:33.858406 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.858385    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 16:02:33.858489 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.858431    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:33.958133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.958084    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:33.959316 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.959050    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:33.959316 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.959087    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:33.959316 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.959099    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:33.959316 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.959127    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-59.ec2.internal"
Jul 28 16:02:33.969330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:33.969309    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-59.ec2.internal"
Jul 28 16:02:33.969443 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.969339    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-59.ec2.internal\": node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:33.986796 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:33.986735    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.007449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.007409    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 16:02:34.008833 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.008811    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 16:02:34.008949 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.008865    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 16:02:34.008949 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.008889    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 16:02:34.008949 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.008926    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 16:02:34.011041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.011022    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:02:34.087102 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.087043    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.109007 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.108978    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"]
Jul 28 16:02:34.109087 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.109054    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:34.110193 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.110173    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:34.110296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.110207    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:34.110296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.110222    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:34.111728 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.111713    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:34.111844 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.111828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.111902 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.111879    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:34.112560 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.112543    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:34.112626 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.112574    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:34.112626 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.112585    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:34.113083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.113064    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:34.113209 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.113093    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:34.113209 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.113106    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:34.113759 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.113744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.113821 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.113771    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:02:34.114446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.114429    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:02:34.114536 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.114463    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:02:34.114536 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.114476    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:02:34.142914 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.142889    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-59.ec2.internal\" not found" node="ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.147674 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.147657    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-59.ec2.internal\" not found" node="ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.187771 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.187736    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.206676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.206647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d42031d4bd857d1b7419a26ede69f8dc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-59.ec2.internal\" (UID: \"d42031d4bd857d1b7419a26ede69f8dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.206747 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.206681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.206747 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.206711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.287945 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.287868    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.307304 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d42031d4bd857d1b7419a26ede69f8dc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-59.ec2.internal\" (UID: \"d42031d4bd857d1b7419a26ede69f8dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.307369 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.307369 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.307369 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.307474 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d42031d4bd857d1b7419a26ede69f8dc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-59.ec2.internal\" (UID: \"d42031d4bd857d1b7419a26ede69f8dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.307474 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.307408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/793213a140d64c21d33c929148a85df8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal\" (UID: \"793213a140d64c21d33c929148a85df8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.389950 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.389898    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.448429 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.448398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.450940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.450913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:34.490911 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.490876    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.594919 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.594819    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.695545 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.695511    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.745913 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.745881    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 16:02:34.746339 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.746066    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:02:34.796397 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.796364    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:34.806060 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.806013    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:02:34.819292 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.819254    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:57:34 +0000 UTC" deadline="2028-02-21 12:27:23.891528032 +0000 UTC"
Jul 28 16:02:34.819292 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:34.819284    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13748h24m49.072247241s"
Jul 28 16:02:34.899275 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:34.899186    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:35.006567 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.006514    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:35.109325 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.109289    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:35.135751 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.135724    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:02:35.210303 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.210225    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:35.310509 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.310471    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-59.ec2.internal\" not found"
Jul 28 16:02:35.371651 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.371624    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:02:35.405279 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.405233    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:35.427744 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.427718    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:02:35.428812 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.428798    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal"
Jul 28 16:02:35.441276 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.441256    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:02:35.787547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.787504    2579 apiserver.go:52] "Watching apiserver"
Jul 28 16:02:35.796464 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.796431    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 16:02:35.798688 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.798650    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-q8vsl","kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal","openshift-dns/node-resolver-dqjnl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal","openshift-multus/multus-additional-cni-plugins-zvgkp","openshift-network-operator/iptables-alerter-crvkz","openshift-ovn-kubernetes/ovnkube-node-rc9lx","kube-system/global-pull-secret-syncer-jmktt","kube-system/konnectivity-agent-xqxd6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd","openshift-cluster-node-tuning-operator/tuned-4xppf","openshift-image-registry/node-ca-c6pdm","openshift-multus/multus-4zhn2","openshift-multus/network-metrics-daemon-ztqjn"]
Jul 28 16:02:35.800851 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.800831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.802438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.802416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.803400 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.803381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 16:02:35.803493 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.803441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.803559 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.803440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5j8jv\""
Jul 28 16:02:35.803559 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.803441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.803726 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.803710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.804505 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.804489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 16:02:35.804674 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.804659    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 16:02:35.805011 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.804993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.805238 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.805064    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:35.805238 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.805238 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 16:02:35.805470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.805619 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4pths\""
Jul 28 16:02:35.805674 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805626    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.805740 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.805894 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.805882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-876sq\""
Jul 28 16:02:35.806428 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.806411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.807748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.807729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:35.808427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.808413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qqwmz\""
Jul 28 16:02:35.808497 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.808464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 16:02:35.808623 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.808608    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 16:02:35.809162 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.809146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:35.809633 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.809614    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.810152 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.810011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6882q\""
Jul 28 16:02:35.810152 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.810034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.810152 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.810018    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 16:02:35.810962 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.810948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.811654 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.811635    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 16:02:35.811744 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.811722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bhgd6\""
Jul 28 16:02:35.812017 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.811980    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 16:02:35.812774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.812759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:35.812855 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.812829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:35.814196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jwqgk\""
Jul 28 16:02:35.814260 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 16:02:35.814321 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 16:02:35.814378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-etc-kubernetes\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814487 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814487 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fb6s\" (UniqueName: \"kubernetes.io/projected/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kube-api-access-2fb6s\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814487 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-system-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814487 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-multus-daemon-config\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-device-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-cni-binary-copy\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk845\" (UniqueName: \"kubernetes.io/projected/0e77f42d-6f86-4e08-b27e-ce9f7323662b-kube-api-access-zk845\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-socket-dir-parent\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814640 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.814629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-netns\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-bin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814727    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-kubelet\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-hostroot\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03033695-3e53-43bc-b5e7-c121d4dd9fe0-tmp-dir\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69cl\" (UniqueName: \"kubernetes.io/projected/03033695-3e53-43bc-b5e7-c121d4dd9fe0-kube-api-access-c69cl\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.814954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-k8s-cni-cncf-io\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.814971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-conf-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815001    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqlz\" (UniqueName: \"kubernetes.io/projected/201d3cf9-6472-422b-b45f-a833061e3916-kube-api-access-qdqlz\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-os-release\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-multus\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-multus-certs\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03033695-3e53-43bc-b5e7-c121d4dd9fe0-hosts-file\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.815530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.815268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-cnibin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.817398 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.817380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.819292 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.819197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:35.819362 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.819298    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx55l\""
Jul 28 16:02:35.819630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.819608    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.819697 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.819672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.821072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.821055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 16:02:35.821371 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.821351    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dxrtv\""
Jul 28 16:02:35.821446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.821370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 16:02:35.821446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.821420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 16:02:35.843675 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.843651    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:02:35.849404 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.849378    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhnl7"
Jul 28 16:02:35.860189 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.860172    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhnl7"
Jul 28 16:02:35.905019 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.904995    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 16:02:35.915819 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.915924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-kubelet-config\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:35.915924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.915924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86b6b215-9318-4e1d-b112-6fb7eb1e5025-host\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:35.916041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-multus-daemon-config\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-var-lib-kubelet\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.916041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-device-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.915993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15f7f1d2-fca9-4360-8b13-3e202faff941-iptables-alerter-script\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:35.916041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-netd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-device-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-netns\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-kubelet\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-netns\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-kubelet\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-slash\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.916291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-dbus\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-node-log\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-k8s-cni-cncf-io\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqlz\" (UniqueName: \"kubernetes.io/projected/201d3cf9-6472-422b-b45f-a833061e3916-kube-api-access-qdqlz\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-modprobe-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-k8s-cni-cncf-io\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rsp\" (UniqueName: \"kubernetes.io/projected/86b6b215-9318-4e1d-b112-6fb7eb1e5025-kube-api-access-s8rsp\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-multus-certs\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15f7f1d2-fca9-4360-8b13-3e202faff941-host-slash\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66ct\" (UniqueName: \"kubernetes.io/projected/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-kube-api-access-p66ct\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-etc-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.916686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-run-multus-certs\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-multus-daemon-config\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.916686    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-sys\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:35.916804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:02:36.416780021 +0000 UTC m=+2.968378009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-cnibin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/292976e5-5217-4dc6-8804-ea7015cb146d-konnectivity-ca\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.916984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-os-release\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-kubelet\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-systemd-units\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-cnibin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-config\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-etc-kubernetes\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.917446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-etc-kubernetes\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-conf\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skv45\" (UniqueName: \"kubernetes.io/projected/5757be10-67b7-44f8-9ad6-ecf44788aeb2-kube-api-access-skv45\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-netns\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e56e3257-fb3a-4cb3-a1da-3cfd14665270-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-bin\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-run\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-systemd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-env-overrides\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.918313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-cni-binary-copy\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk845\" (UniqueName: \"kubernetes.io/projected/0e77f42d-6f86-4e08-b27e-ce9f7323662b-kube-api-access-zk845\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmb8\" (UniqueName: \"kubernetes.io/projected/15f7f1d2-fca9-4360-8b13-3e202faff941-kube-api-access-qmmb8\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-host\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-socket-dir-parent\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-socket-dir-parent\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-bin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-hostroot\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.917951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03033695-3e53-43bc-b5e7-c121d4dd9fe0-tmp-dir\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-bin\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-hostroot\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69cl\" (UniqueName: \"kubernetes.io/projected/03033695-3e53-43bc-b5e7-c121d4dd9fe0-kube-api-access-c69cl\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/292976e5-5217-4dc6-8804-ea7015cb146d-agent-certs\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-tuned\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-conf-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/201d3cf9-6472-422b-b45f-a833061e3916-cni-binary-copy\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcqz\" (UniqueName: \"kubernetes.io/projected/1837f764-cb2c-42d2-8a54-40f23c0c1e30-kube-api-access-vgcqz\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-lib-modules\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-multus-conf-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-tmp\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-os-release\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03033695-3e53-43bc-b5e7-c121d4dd9fe0-tmp-dir\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-multus\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-os-release\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-host-var-lib-cni-multus\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03033695-3e53-43bc-b5e7-c121d4dd9fe0-hosts-file\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-var-lib-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cnibin\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03033695-3e53-43bc-b5e7-c121d4dd9fe0-hosts-file\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-ovn\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysconfig\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.919669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-systemd\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86b6b215-9318-4e1d-b112-6fb7eb1e5025-serviceca\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-log-socket\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-script-lib\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-kubernetes\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fb6s\" (UniqueName: \"kubernetes.io/projected/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kube-api-access-2fb6s\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-system-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.920216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.918815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/201d3cf9-6472-422b-b45f-a833061e3916-system-cni-dir\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:35.928782 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.928753    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 16:02:35.932299 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.932266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69cl\" (UniqueName: \"kubernetes.io/projected/03033695-3e53-43bc-b5e7-c121d4dd9fe0-kube-api-access-c69cl\") pod \"node-resolver-dqjnl\" (UID: \"03033695-3e53-43bc-b5e7-c121d4dd9fe0\") " pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:35.932299 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.932286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk845\" (UniqueName: \"kubernetes.io/projected/0e77f42d-6f86-4e08-b27e-ce9f7323662b-kube-api-access-zk845\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:35.932433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.932266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fb6s\" (UniqueName: \"kubernetes.io/projected/e56e3257-fb3a-4cb3-a1da-3cfd14665270-kube-api-access-2fb6s\") pod \"aws-ebs-csi-driver-node-ptxrd\" (UID: \"e56e3257-fb3a-4cb3-a1da-3cfd14665270\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:35.932433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:35.932288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqlz\" (UniqueName: \"kubernetes.io/projected/201d3cf9-6472-422b-b45f-a833061e3916-kube-api-access-qdqlz\") pod \"multus-4zhn2\" (UID: \"201d3cf9-6472-422b-b45f-a833061e3916\") " pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:36.020486 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-ovn\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysconfig\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.020655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-systemd\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.020655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86b6b215-9318-4e1d-b112-6fb7eb1e5025-serviceca\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.020655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-log-socket\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-script-lib\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysconfig\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-kubernetes\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-ovn\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-kubernetes\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.020940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-log-socket\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.020957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-kubelet-config\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-kubelet-config\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86b6b215-9318-4e1d-b112-6fb7eb1e5025-host\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-systemd\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-var-lib-kubelet\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.021435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86b6b215-9318-4e1d-b112-6fb7eb1e5025-serviceca\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15f7f1d2-fca9-4360-8b13-3e202faff941-iptables-alerter-script\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-netd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-var-lib-kubelet\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-slash\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-slash\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-dbus\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86b6b215-9318-4e1d-b112-6fb7eb1e5025-host\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-node-log\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.021822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-node-log\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022206 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8109b844-aca8-4f61-b52c-61dc7132a484-dbus\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.022206 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.021883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-netd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-modprobe-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.022639 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-modprobe-d\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.022639 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rsp\" (UniqueName: \"kubernetes.io/projected/86b6b215-9318-4e1d-b112-6fb7eb1e5025-kube-api-access-s8rsp\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.022757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15f7f1d2-fca9-4360-8b13-3e202faff941-host-slash\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.022757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66ct\" (UniqueName: \"kubernetes.io/projected/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-kube-api-access-p66ct\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.022757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15f7f1d2-fca9-4360-8b13-3e202faff941-host-slash\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.022757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-etc-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15f7f1d2-fca9-4360-8b13-3e202faff941-iptables-alerter-script\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-etc-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.022834    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-sys\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-script-lib\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/292976e5-5217-4dc6-8804-ea7015cb146d-konnectivity-ca\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.022918    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:36.522903046 +0000 UTC m=+3.074501023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-sys\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-os-release\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-kubelet\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.022987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.022992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-systemd-units\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-config\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-kubelet\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-os-release\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-conf\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-systemd-units\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skv45\" (UniqueName: \"kubernetes.io/projected/5757be10-67b7-44f8-9ad6-ecf44788aeb2-kube-api-access-skv45\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-sysctl-conf\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-netns\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-system-cni-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-bin\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.023592 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-run\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-cni-bin\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-systemd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-env-overrides\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-host-run-netns\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-run\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmb8\" (UniqueName: \"kubernetes.io/projected/15f7f1d2-fca9-4360-8b13-3e202faff941-kube-api-access-qmmb8\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-host\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/292976e5-5217-4dc6-8804-ea7015cb146d-agent-certs\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/292976e5-5217-4dc6-8804-ea7015cb146d-konnectivity-ca\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-tuned\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcqz\" (UniqueName: \"kubernetes.io/projected/1837f764-cb2c-42d2-8a54-40f23c0c1e30-kube-api-access-vgcqz\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-run-systemd\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-lib-modules\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-tmp\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-host\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.024450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovnkube-config\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-var-lib-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cnibin\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cnibin\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1837f764-cb2c-42d2-8a54-40f23c0c1e30-env-overrides\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.023891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5757be10-67b7-44f8-9ad6-ecf44788aeb2-lib-modules\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.024067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1837f764-cb2c-42d2-8a54-40f23c0c1e30-var-lib-openvswitch\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.024222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.024368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.025095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.024611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1837f764-cb2c-42d2-8a54-40f23c0c1e30-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.026033 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.026017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-tmp\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.026200 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.026184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5757be10-67b7-44f8-9ad6-ecf44788aeb2-etc-tuned\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.026275 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.026259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/292976e5-5217-4dc6-8804-ea7015cb146d-agent-certs\") pod \"konnectivity-agent-xqxd6\" (UID: \"292976e5-5217-4dc6-8804-ea7015cb146d\") " pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:36.031893 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.031876    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:36.031893 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.031894    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:36.032029 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.031905    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:36.032029 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.031974    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:36.531961583 +0000 UTC m=+3.083559559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:36.034893 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.034865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66ct\" (UniqueName: \"kubernetes.io/projected/4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f-kube-api-access-p66ct\") pod \"multus-additional-cni-plugins-zvgkp\" (UID: \"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f\") " pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.036342 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.036314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skv45\" (UniqueName: \"kubernetes.io/projected/5757be10-67b7-44f8-9ad6-ecf44788aeb2-kube-api-access-skv45\") pod \"tuned-4xppf\" (UID: \"5757be10-67b7-44f8-9ad6-ecf44788aeb2\") " pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.036599 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.036576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rsp\" (UniqueName: \"kubernetes.io/projected/86b6b215-9318-4e1d-b112-6fb7eb1e5025-kube-api-access-s8rsp\") pod \"node-ca-c6pdm\" (UID: \"86b6b215-9318-4e1d-b112-6fb7eb1e5025\") " pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.036868 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.036850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmb8\" (UniqueName: \"kubernetes.io/projected/15f7f1d2-fca9-4360-8b13-3e202faff941-kube-api-access-qmmb8\") pod \"iptables-alerter-crvkz\" (UID: \"15f7f1d2-fca9-4360-8b13-3e202faff941\") " pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.038467 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.038405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcqz\" (UniqueName: \"kubernetes.io/projected/1837f764-cb2c-42d2-8a54-40f23c0c1e30-kube-api-access-vgcqz\") pod \"ovnkube-node-rc9lx\" (UID: \"1837f764-cb2c-42d2-8a54-40f23c0c1e30\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.119365 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.119323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd"
Jul 28 16:02:36.138218 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.138196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zhn2"
Jul 28 16:02:36.146789 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.146764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqjnl"
Jul 28 16:02:36.160266 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.160247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvgkp"
Jul 28 16:02:36.178849 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.178823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crvkz"
Jul 28 16:02:36.193012 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.192987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:02:36.214264 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.214235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:02:36.232850 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.232824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xppf"
Jul 28 16:02:36.236327 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.236306    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:02:36.243119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.243102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c6pdm"
Jul 28 16:02:36.426311 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.426222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:36.426449 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.426370    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:36.426449 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.426435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:02:37.426421573 +0000 UTC m=+3.978019546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:36.527444 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.527404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:36.527647 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.527513    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:36.527647 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.527577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:37.527560165 +0000 UTC m=+4.079158138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:36.628395 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.628360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:36.628696 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.628580    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:36.628696 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.628604    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:36.628696 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.628614    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:36.628696 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:36.628676    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:37.628659829 +0000 UTC m=+4.180257802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:36.729815 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.729779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201d3cf9_6472_422b_b45f_a833061e3916.slice/crio-27a077e1e3239bff9fe33f732ac036c20fe026eac48806636c4e2f7c262debab WatchSource:0}: Error finding container 27a077e1e3239bff9fe33f732ac036c20fe026eac48806636c4e2f7c262debab: Status 404 returned error can't find the container with id 27a077e1e3239bff9fe33f732ac036c20fe026eac48806636c4e2f7c262debab
Jul 28 16:02:36.730546 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.730516    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0c4aa6_0a2d_4fa6_9746_9ac4c315af4f.slice/crio-28e70dd9136eb9c5301e2ccb15f34599455969d923700b02a6ae03f2d10385ae WatchSource:0}: Error finding container 28e70dd9136eb9c5301e2ccb15f34599455969d923700b02a6ae03f2d10385ae: Status 404 returned error can't find the container with id 28e70dd9136eb9c5301e2ccb15f34599455969d923700b02a6ae03f2d10385ae
Jul 28 16:02:36.741810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.741791    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:02:36.750269 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.750246    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56e3257_fb3a_4cb3_a1da_3cfd14665270.slice/crio-d0eb1a1565cd5be9ee77473e9b856f5980f31f950af9144461779676f138c87f WatchSource:0}: Error finding container d0eb1a1565cd5be9ee77473e9b856f5980f31f950af9144461779676f138c87f: Status 404 returned error can't find the container with id d0eb1a1565cd5be9ee77473e9b856f5980f31f950af9144461779676f138c87f
Jul 28 16:02:36.754091 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.754067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1837f764_cb2c_42d2_8a54_40f23c0c1e30.slice/crio-512dd4693d158fcd1e2c043291df8bd923b76bf445d5320b33dd3021473041ea WatchSource:0}: Error finding container 512dd4693d158fcd1e2c043291df8bd923b76bf445d5320b33dd3021473041ea: Status 404 returned error can't find the container with id 512dd4693d158fcd1e2c043291df8bd923b76bf445d5320b33dd3021473041ea
Jul 28 16:02:36.759403 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.759380    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b6b215_9318_4e1d_b112_6fb7eb1e5025.slice/crio-55704868c176d5dfbf16fe1dace1a5cfc4538320cffff310eda771822989d0c2 WatchSource:0}: Error finding container 55704868c176d5dfbf16fe1dace1a5cfc4538320cffff310eda771822989d0c2: Status 404 returned error can't find the container with id 55704868c176d5dfbf16fe1dace1a5cfc4538320cffff310eda771822989d0c2
Jul 28 16:02:36.762705 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.762681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f7f1d2_fca9_4360_8b13_3e202faff941.slice/crio-b8e10ed0d57c639f454f5764ebaac9a497745cc773339e8cc322a30eb7edf534 WatchSource:0}: Error finding container b8e10ed0d57c639f454f5764ebaac9a497745cc773339e8cc322a30eb7edf534: Status 404 returned error can't find the container with id b8e10ed0d57c639f454f5764ebaac9a497745cc773339e8cc322a30eb7edf534
Jul 28 16:02:36.765540 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.765523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757be10_67b7_44f8_9ad6_ecf44788aeb2.slice/crio-3e874ce495b18ec4c03a62cdbdc926766ad50df74db83b204bbdb5bb081dc124 WatchSource:0}: Error finding container 3e874ce495b18ec4c03a62cdbdc926766ad50df74db83b204bbdb5bb081dc124: Status 404 returned error can't find the container with id 3e874ce495b18ec4c03a62cdbdc926766ad50df74db83b204bbdb5bb081dc124
Jul 28 16:02:36.787839 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.787819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03033695_3e53_43bc_b5e7_c121d4dd9fe0.slice/crio-9ba536cdecb8efdcbbb2880398ebb47e3c570d71052c0480aeae99836e71049e WatchSource:0}: Error finding container 9ba536cdecb8efdcbbb2880398ebb47e3c570d71052c0480aeae99836e71049e: Status 404 returned error can't find the container with id 9ba536cdecb8efdcbbb2880398ebb47e3c570d71052c0480aeae99836e71049e
Jul 28 16:02:36.862546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.862507    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:57:35 +0000 UTC" deadline="2028-02-26 08:06:37.155612174 +0000 UTC"
Jul 28 16:02:36.862546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:36.862537    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13864h4m0.293078911s"
Jul 28 16:02:36.938518 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.938492    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793213a140d64c21d33c929148a85df8.slice/crio-8675fc45c032d722b4b7d4f8280af6a104dedd19d78066c8c1b64716c03c01cf WatchSource:0}: Error finding container 8675fc45c032d722b4b7d4f8280af6a104dedd19d78066c8c1b64716c03c01cf: Status 404 returned error can't find the container with id 8675fc45c032d722b4b7d4f8280af6a104dedd19d78066c8c1b64716c03c01cf
Jul 28 16:02:36.971169 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:02:36.971135    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42031d4bd857d1b7419a26ede69f8dc.slice/crio-885440155ed40cb808369148a0255d65be83698861cc03a330399b92ba33a5ca WatchSource:0}: Error finding container 885440155ed40cb808369148a0255d65be83698861cc03a330399b92ba33a5ca: Status 404 returned error can't find the container with id 885440155ed40cb808369148a0255d65be83698861cc03a330399b92ba33a5ca
Jul 28 16:02:37.010155 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.010125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:37.010155 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.010150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:37.010372 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.010225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:37.010415 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.010363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:37.016029 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.015984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"512dd4693d158fcd1e2c043291df8bd923b76bf445d5320b33dd3021473041ea"}
Jul 28 16:02:37.017099 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.017075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd" event={"ID":"e56e3257-fb3a-4cb3-a1da-3cfd14665270","Type":"ContainerStarted","Data":"d0eb1a1565cd5be9ee77473e9b856f5980f31f950af9144461779676f138c87f"}
Jul 28 16:02:37.018142 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.018120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerStarted","Data":"28e70dd9136eb9c5301e2ccb15f34599455969d923700b02a6ae03f2d10385ae"}
Jul 28 16:02:37.019096 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.019077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal" event={"ID":"d42031d4bd857d1b7419a26ede69f8dc","Type":"ContainerStarted","Data":"885440155ed40cb808369148a0255d65be83698861cc03a330399b92ba33a5ca"}
Jul 28 16:02:37.020221 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.020201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal" event={"ID":"793213a140d64c21d33c929148a85df8","Type":"ContainerStarted","Data":"8675fc45c032d722b4b7d4f8280af6a104dedd19d78066c8c1b64716c03c01cf"}
Jul 28 16:02:37.021283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.021265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqjnl" event={"ID":"03033695-3e53-43bc-b5e7-c121d4dd9fe0","Type":"ContainerStarted","Data":"9ba536cdecb8efdcbbb2880398ebb47e3c570d71052c0480aeae99836e71049e"}
Jul 28 16:02:37.022293 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.022273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c6pdm" event={"ID":"86b6b215-9318-4e1d-b112-6fb7eb1e5025","Type":"ContainerStarted","Data":"55704868c176d5dfbf16fe1dace1a5cfc4538320cffff310eda771822989d0c2"}
Jul 28 16:02:37.023379 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.023361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zhn2" event={"ID":"201d3cf9-6472-422b-b45f-a833061e3916","Type":"ContainerStarted","Data":"27a077e1e3239bff9fe33f732ac036c20fe026eac48806636c4e2f7c262debab"}
Jul 28 16:02:37.024374 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.024354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xppf" event={"ID":"5757be10-67b7-44f8-9ad6-ecf44788aeb2","Type":"ContainerStarted","Data":"3e874ce495b18ec4c03a62cdbdc926766ad50df74db83b204bbdb5bb081dc124"}
Jul 28 16:02:37.025485 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.025466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crvkz" event={"ID":"15f7f1d2-fca9-4360-8b13-3e202faff941","Type":"ContainerStarted","Data":"b8e10ed0d57c639f454f5764ebaac9a497745cc773339e8cc322a30eb7edf534"}
Jul 28 16:02:37.434908 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.434819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:37.435062 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.435006    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:37.435133 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.435071    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:02:39.435054041 +0000 UTC m=+5.986652027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:37.535794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.535265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:37.535794 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.535397    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:37.535794 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.535473    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:39.535451533 +0000 UTC m=+6.087049524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:37.636315 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.636279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:37.636492 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.636479    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:37.636492 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.636499    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:37.636610 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.636511    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:37.636610 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:37.636566    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:39.63654901 +0000 UTC m=+6.188146984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:37.862951 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.862842    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:57:35 +0000 UTC" deadline="2028-01-30 13:14:17.608663336 +0000 UTC"
Jul 28 16:02:37.862951 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:37.862900    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13221h11m39.745767865s"
Jul 28 16:02:38.010043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:38.009451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:38.010043 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:38.009624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:39.009582 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:39.009544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:39.010263 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.009682    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:39.010263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:39.010120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:39.010263 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.010208    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:39.459653 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:39.459557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:39.459811 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.459738    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:39.459811 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.459802    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:02:43.45978438 +0000 UTC m=+10.011382370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:39.561099 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:39.561055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:39.561283 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.561213    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:39.561334 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.561293    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:43.561272077 +0000 UTC m=+10.112870063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:39.661534 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:39.661490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:39.661713 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.661702    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:39.661780 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.661725    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:39.661780 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.661738    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:39.661916 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:39.661797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:43.66177854 +0000 UTC m=+10.213376532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:40.009251 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:40.009210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:40.009428 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:40.009376    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:41.009776 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:41.009119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:41.009776 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:41.009191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:41.009776 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:41.009289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:41.009776 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:41.009714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:42.009867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:42.009818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:42.010317 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:42.009981    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:43.010381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:43.010082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:43.010381 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.010242    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:43.010381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:43.010255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:43.010381 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.010346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:43.494314 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:43.494188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:43.499267 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.499142    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:43.499267 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.499227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:02:51.499206517 +0000 UTC m=+18.050804493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:43.595230 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:43.595188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:43.595952 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.595415    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:43.595952 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.595490    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:51.59546959 +0000 UTC m=+18.147067576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:43.696526 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:43.696464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:43.696726 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.696609    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:43.696726 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.696626    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:43.696726 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.696636    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:43.696726 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:43.696688    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:51.696675098 +0000 UTC m=+18.248273083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:44.011720 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:44.011247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:44.011720 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:44.011369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:45.009254 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:45.009152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:45.009254 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:45.009197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:45.009499 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:45.009295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:45.009499 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:45.009438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:46.009697 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:46.009661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:46.010094 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:46.009824    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:47.009834 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:47.009783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:47.010285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:47.009783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:47.010285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:47.009957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:47.010285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:47.010038    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:48.009500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:48.009463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:48.009692 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:48.009613    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:49.009756 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:49.009722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:49.010221 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:49.009836    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:49.010221 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:49.009896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:49.010221 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:49.010009    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:50.009439 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:50.009403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:50.009639 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:50.009549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:51.009098 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:51.009061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:51.009562 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:51.009061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:51.009562 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.009207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:51.009562 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.009283    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:51.557548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:51.557520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:51.557727 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.557651    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:51.557727 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.557709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:03:07.557693421 +0000 UTC m=+34.109291396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:02:51.657925 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:51.657889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:51.658091 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.658052    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:51.658143 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.658130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:07.658108818 +0000 UTC m=+34.209706792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:02:51.758787 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:51.758755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:51.758964 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.758905    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:02:51.758964 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.758926    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:02:51.758964 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.758937    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:51.759129 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:51.759001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:03:07.758982177 +0000 UTC m=+34.310580171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:02:52.021986 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:52.021908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:52.022401 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:52.022047    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:53.009970 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:53.009936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:53.010245 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:53.010051    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:53.010245 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:53.010109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:53.010245 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:53.010218    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:54.010503 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:54.010309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:54.010976 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:54.010592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:55.009257 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:55.009221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:55.009408 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:55.009333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:55.009408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:55.009382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:55.009502 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:55.009465    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:56.009975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.009712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:56.010640 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:56.010097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:56.074826 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.074794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"e49fa11a9ee1acf5ecbafb239e17a1fbbacb8f26b8ac56af142a2248871ab60a"}
Jul 28 16:02:56.074942 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.074925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"164b98a45e91c5fec52201bcacb673d4dbe1a5ad956f2ad8a10ac2930e51c591"}
Jul 28 16:02:56.075002 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.074986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"ad1d89c5c86f40eadd9a45cd7961a58209209ecb484c248af191b6aa658dd622"}
Jul 28 16:02:56.075058 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.075045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"6365cd0e83b70c0b4c3a8531b3267f95198e163886d00a455d3c0f78d5abd45d"}
Jul 28 16:02:56.076147 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.076118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd" event={"ID":"e56e3257-fb3a-4cb3-a1da-3cfd14665270","Type":"ContainerStarted","Data":"e309fc75ae2da1ddc7bb76728a1a6227f87a1856505b1a645ddeaf2430b43234"}
Jul 28 16:02:56.077561 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.077539    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="67dd17adf52657981faa9bb59ba5e0df16b536a32eaf774fde945d6a1415a3a7" exitCode=0
Jul 28 16:02:56.077648 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.077599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"67dd17adf52657981faa9bb59ba5e0df16b536a32eaf774fde945d6a1415a3a7"}
Jul 28 16:02:56.079204 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.079184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal" event={"ID":"d42031d4bd857d1b7419a26ede69f8dc","Type":"ContainerStarted","Data":"2600974a47969f742323f156a5fe2839af962110a330d8a587e0b2ab537fb6e3"}
Jul 28 16:02:56.086612 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.086339    2579 generic.go:356] "Generic (PLEG): container finished" podID="793213a140d64c21d33c929148a85df8" containerID="2a3035c6be420bdf8de719097b583c81fd7ec3d3ed94e4f32c8984f48970ea9f" exitCode=0
Jul 28 16:02:56.086612 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.086412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal" event={"ID":"793213a140d64c21d33c929148a85df8","Type":"ContainerDied","Data":"2a3035c6be420bdf8de719097b583c81fd7ec3d3ed94e4f32c8984f48970ea9f"}
Jul 28 16:02:56.088364 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.088339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqjnl" event={"ID":"03033695-3e53-43bc-b5e7-c121d4dd9fe0","Type":"ContainerStarted","Data":"22956813b9a0772640ee7ab2b819ee028bdda6fface6243b0be766e667fde831"}
Jul 28 16:02:56.091275 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.091252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c6pdm" event={"ID":"86b6b215-9318-4e1d-b112-6fb7eb1e5025","Type":"ContainerStarted","Data":"f413587e0faa2532fbacafcdead30b6faed99d3781f7d35447a41b515ec21cb6"}
Jul 28 16:02:56.092885 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.092840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zhn2" event={"ID":"201d3cf9-6472-422b-b45f-a833061e3916","Type":"ContainerStarted","Data":"a6253b17b9d685a0a6d9e4b1a31834aff007b4c46f736cde3b640dfe5ada6092"}
Jul 28 16:02:56.094060 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.094038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqxd6" event={"ID":"292976e5-5217-4dc6-8804-ea7015cb146d","Type":"ContainerStarted","Data":"a20542d4f7eb198765c9fee0f6d74d9af0121fcd80467d66ff8b38729fcd4679"}
Jul 28 16:02:56.096026 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.095999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xppf" event={"ID":"5757be10-67b7-44f8-9ad6-ecf44788aeb2","Type":"ContainerStarted","Data":"50f47ad927ae7975099284944385f81c98e479a0f2677c858a5c69c33ba61eac"}
Jul 28 16:02:56.115629 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.115394    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c6pdm" podStartSLOduration=3.457776106 podStartE2EDuration="22.115374216s" totalImagesPullingTime="18.65759811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:56.1141716 +0000 UTC m=+22.665769595" watchObservedRunningTime="2026-07-28 16:02:56.115374216 +0000 UTC m=+22.666972213"
Jul 28 16:02:56.155466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.155246    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4xppf" podStartSLOduration=3.500620614 podStartE2EDuration="22.155225095s" totalImagesPullingTime="18.654604481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:56.155138921 +0000 UTC m=+22.706736939" watchObservedRunningTime="2026-07-28 16:02:56.155225095 +0000 UTC m=+22.706823091"
Jul 28 16:02:56.171324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.170964    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-59.ec2.internal" podStartSLOduration=21.170936235 podStartE2EDuration="21.170936235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:56.170233463 +0000 UTC m=+22.721831455" watchObservedRunningTime="2026-07-28 16:02:56.170936235 +0000 UTC m=+22.722534230"
Jul 28 16:02:56.192598 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.192533    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4zhn2" podStartSLOduration=3.489840686 podStartE2EDuration="22.192513127s" totalImagesPullingTime="18.702672441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:56.191774671 +0000 UTC m=+22.743372664" watchObservedRunningTime="2026-07-28 16:02:56.192513127 +0000 UTC m=+22.744111122"
Jul 28 16:02:56.210324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:56.210276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dqjnl" podStartSLOduration=3.577150615 podStartE2EDuration="22.210260371s" totalImagesPullingTime="18.633109756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:56.209978822 +0000 UTC m=+22.761576815" watchObservedRunningTime="2026-07-28 16:02:56.210260371 +0000 UTC m=+22.761858364"
Jul 28 16:02:57.009448 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.009206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:57.009631 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.009252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:57.009631 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:57.009549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:57.009755 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:57.009636    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:57.101096 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.101061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal" event={"ID":"793213a140d64c21d33c929148a85df8","Type":"ContainerStarted","Data":"987e1ff65456d396495ac22dcbfdb91b6d2435b285bf048c8c21b2bc09014435"}
Jul 28 16:02:57.102842 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.102814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crvkz" event={"ID":"15f7f1d2-fca9-4360-8b13-3e202faff941","Type":"ContainerStarted","Data":"02cd7374f123280c74ccd14462c2a1327ab10605333aadc6898c895a88a22d8c"}
Jul 28 16:02:57.106080 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.106052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"73c4e627a8cd9505a284856c8b0a8bc46a7252a63b4cfada542ee43935bf4609"}
Jul 28 16:02:57.106220 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.106138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"4101cd2aea10e45f18fef955e62e2a13c69aec6647c189ed341ff7d67fbfc19f"}
Jul 28 16:02:57.118771 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.118728    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-59.ec2.internal" podStartSLOduration=22.118715807 podStartE2EDuration="22.118715807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:57.118237139 +0000 UTC m=+23.669835131" watchObservedRunningTime="2026-07-28 16:02:57.118715807 +0000 UTC m=+23.670313801"
Jul 28 16:02:57.133576 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.133533    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crvkz" podStartSLOduration=8.819396232 podStartE2EDuration="23.133519783s" totalImagesPullingTime="14.314123551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:57.133414427 +0000 UTC m=+23.685012421" watchObservedRunningTime="2026-07-28 16:02:57.133519783 +0000 UTC m=+23.685117782"
Jul 28 16:02:57.306610 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.306570    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 16:02:57.876401 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.876249    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T16:02:57.30659396Z","UUID":"7ed31732-566c-4c4b-bf07-0ff9514deb7a","Handler":null,"Name":"","Endpoint":""}
Jul 28 16:02:57.877970 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.877946    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 16:02:57.878116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:57.877989    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 16:02:58.009658 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:58.009623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:02:58.009871 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:58.009825    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:02:58.110168 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:58.110130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqxd6" event={"ID":"292976e5-5217-4dc6-8804-ea7015cb146d","Type":"ContainerStarted","Data":"ef3465ee21ca9356cf73ce9b7dcacc47c6907354789d0c57fcc58ac07f24b116"}
Jul 28 16:02:58.112104 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:58.112069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd" event={"ID":"e56e3257-fb3a-4cb3-a1da-3cfd14665270","Type":"ContainerStarted","Data":"139b1417f3d2e3e5a5bbbd93d59689cdbfcc1a95b69b0f9edff7a879a6f2d590"}
Jul 28 16:02:58.128281 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:58.128195    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xqxd6" podStartSLOduration=22.364163582 podStartE2EDuration="24.128176049s" totalImagesPullingTime="1.764012467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:58.127325297 +0000 UTC m=+24.678923290" watchObservedRunningTime="2026-07-28 16:02:58.128176049 +0000 UTC m=+24.679774043"
Jul 28 16:02:59.009416 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:59.009383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:02:59.009686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:59.009428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:02:59.009686 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:59.009502    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:02:59.009686 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:02:59.009652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:02:59.118594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:59.118555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"266dbb523eb0b0b1923719cf9c93c6da995d55d5c27d6f3c6041ae69cb2622dd"}
Jul 28 16:02:59.120788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:59.120758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd" event={"ID":"e56e3257-fb3a-4cb3-a1da-3cfd14665270","Type":"ContainerStarted","Data":"6c80fb4f0cff47772f02e8a40642687d711f3c15fc057d0c324ff49e22955e66"}
Jul 28 16:02:59.139137 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:02:59.139088    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxrd" podStartSLOduration=8.56824997 podStartE2EDuration="25.139066932s" totalImagesPullingTime="16.570816962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:59.138469539 +0000 UTC m=+25.690067533" watchObservedRunningTime="2026-07-28 16:02:59.139066932 +0000 UTC m=+25.690664929"
Jul 28 16:03:00.009625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:00.009587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:00.009842 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:00.009715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:03:00.354602 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:00.354527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:03:01.009947 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.009718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:01.010098 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.009718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:01.010098 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:01.009978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:03:01.010098 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:01.010050    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:03:01.040992 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.040967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:03:01.041633 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.041617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:03:01.128912 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.128875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" event={"ID":"1837f764-cb2c-42d2-8a54-40f23c0c1e30","Type":"ContainerStarted","Data":"fd1e46dd5c75c6e69314a32bd3088ede2a8c99995076ca1afec0f8b698b92cec"}
Jul 28 16:03:01.129226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.129189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:01.130821 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.130796    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="da9984549077325d50d6da0fd75e37cf48e722c081a63c37f1bcdcd63439588a" exitCode=0
Jul 28 16:03:01.130920 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.130876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"da9984549077325d50d6da0fd75e37cf48e722c081a63c37f1bcdcd63439588a"}
Jul 28 16:03:01.131539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.131525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xqxd6"
Jul 28 16:03:01.144734 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.144715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:01.160322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:01.160283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx" podStartSLOduration=8.253042817 podStartE2EDuration="27.160271959s" totalImagesPullingTime="18.907229142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:01.159652313 +0000 UTC m=+27.711250305" watchObservedRunningTime="2026-07-28 16:03:01.160271959 +0000 UTC m=+27.711869951"
Jul 28 16:03:02.009310 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.009232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:02.009779 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:02.009348    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:03:02.133230 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.133196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:02.133370 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.133263    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:02.147183 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.147160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:02.923409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.923381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8vsl"]
Jul 28 16:03:02.923564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.923499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:02.923643 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:02.923600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:03:02.926910 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.926886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jmktt"]
Jul 28 16:03:02.927035 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.926974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:02.927072 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:02.927048    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:03:02.927602 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.927557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ztqjn"]
Jul 28 16:03:02.927666 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:02.927654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:02.927782 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:02.927763    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:03:04.010056 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:04.010018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:04.010712 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:04.010123    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:03:05.010037 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:05.009841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:05.010197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:05.009868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:05.010197 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:05.010120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:03:05.010513 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:05.010204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:03:06.009230 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:06.009195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:06.009413 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:06.009359    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:03:07.009758 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.009725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:07.010144 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.009827    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8vsl" podUID="7caceb3f-a665-4ef7-bde8-d714d0f762fb"
Jul 28 16:03:07.010144 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.009894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:07.010144 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.009999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztqjn" podUID="0e77f42d-6f86-4e08-b27e-ce9f7323662b"
Jul 28 16:03:07.145193 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.145164    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="8a6b36e5efdbdfdb0f9670759e6375285c023771bd8f7cf5e7792494367e1f08" exitCode=0
Jul 28 16:03:07.145335 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.145199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"8a6b36e5efdbdfdb0f9670759e6375285c023771bd8f7cf5e7792494367e1f08"}
Jul 28 16:03:07.576208 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.576168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:07.576373 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.576348    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:03:07.576449 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.576442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:03:39.576424286 +0000 UTC m=+66.128022263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:03:07.676763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.676731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:07.676939 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.676866    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:03:07.676939 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.676935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret podName:8109b844-aca8-4f61-b52c-61dc7132a484 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:39.676915945 +0000 UTC m=+66.228513930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret") pod "global-pull-secret-syncer-jmktt" (UID: "8109b844-aca8-4f61-b52c-61dc7132a484") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:03:07.778052 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:07.778017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:07.778223 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.778158    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:03:07.778223 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.778176    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:03:07.778223 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.778187    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rr8cj for pod openshift-network-diagnostics/network-check-target-q8vsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:03:07.778370 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:07.778251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj podName:7caceb3f-a665-4ef7-bde8-d714d0f762fb nodeName:}" failed. No retries permitted until 2026-07-28 16:03:39.778231792 +0000 UTC m=+66.329829791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8cj" (UniqueName: "kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj") pod "network-check-target-q8vsl" (UID: "7caceb3f-a665-4ef7-bde8-d714d0f762fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:03:08.009843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.009813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:08.010226 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:08.009940    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmktt" podUID="8109b844-aca8-4f61-b52c-61dc7132a484"
Jul 28 16:03:08.297087 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.297065    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-59.ec2.internal" event="NodeReady"
Jul 28 16:03:08.297200 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.297190    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 16:03:08.352534 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.352503    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5srrc"]
Jul 28 16:03:08.364544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.364518    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ph7dx"]
Jul 28 16:03:08.364665 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.364649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.366969 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.366951    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 16:03:08.367058 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.367020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ssj24\""
Jul 28 16:03:08.367107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.367073    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 16:03:08.376611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.376584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5srrc"]
Jul 28 16:03:08.376703 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.376623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ph7dx"]
Jul 28 16:03:08.376740 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.376725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.379014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.378962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 16:03:08.379014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.378962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bjngx\""
Jul 28 16:03:08.379172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.379036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 16:03:08.379172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.379097    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 16:03:08.482386 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.482386 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55sq\" (UniqueName: \"kubernetes.io/projected/cb4d19b7-6b21-462a-9db2-11febdef81e4-kube-api-access-w55sq\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.482580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d57eebac-dec7-4853-b9e0-420e03e1bd5c-cert\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.482580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d19b7-6b21-462a-9db2-11febdef81e4-tmp-dir\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.482580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfkn\" (UniqueName: \"kubernetes.io/projected/d57eebac-dec7-4853-b9e0-420e03e1bd5c-kube-api-access-dcfkn\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.482580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.482557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4d19b7-6b21-462a-9db2-11febdef81e4-config-volume\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.583711 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.583624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d19b7-6b21-462a-9db2-11febdef81e4-tmp-dir\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.583711 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.583667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfkn\" (UniqueName: \"kubernetes.io/projected/d57eebac-dec7-4853-b9e0-420e03e1bd5c-kube-api-access-dcfkn\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.583898 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.583863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4d19b7-6b21-462a-9db2-11febdef81e4-config-volume\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.583992 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.583973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.584039 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.583997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d19b7-6b21-462a-9db2-11febdef81e4-tmp-dir\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.584039 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.584007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55sq\" (UniqueName: \"kubernetes.io/projected/cb4d19b7-6b21-462a-9db2-11febdef81e4-kube-api-access-w55sq\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.584117 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.584037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d57eebac-dec7-4853-b9e0-420e03e1bd5c-cert\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.584117 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:08.584079    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:08.584177 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:08.584128    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:09.084114928 +0000 UTC m=+35.635712899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:08.584458 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.584437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4d19b7-6b21-462a-9db2-11febdef81e4-config-volume\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.588083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.588062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d57eebac-dec7-4853-b9e0-420e03e1bd5c-cert\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.595833 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.595804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfkn\" (UniqueName: \"kubernetes.io/projected/d57eebac-dec7-4853-b9e0-420e03e1bd5c-kube-api-access-dcfkn\") pod \"ingress-canary-ph7dx\" (UID: \"d57eebac-dec7-4853-b9e0-420e03e1bd5c\") " pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.595945 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.595869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55sq\" (UniqueName: \"kubernetes.io/projected/cb4d19b7-6b21-462a-9db2-11febdef81e4-kube-api-access-w55sq\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:08.684699 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.684663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ph7dx"
Jul 28 16:03:08.890580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:08.890340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ph7dx"]
Jul 28 16:03:08.895420 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:08.895396    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57eebac_dec7_4853_b9e0_420e03e1bd5c.slice/crio-16316d9c415dd60be2ae22da096af1474dfffaa586152947b741d7b8372dc8b0 WatchSource:0}: Error finding container 16316d9c415dd60be2ae22da096af1474dfffaa586152947b741d7b8372dc8b0: Status 404 returned error can't find the container with id 16316d9c415dd60be2ae22da096af1474dfffaa586152947b741d7b8372dc8b0
Jul 28 16:03:09.009985 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.009951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:09.010415 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.009956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:09.012613 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.012579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:03:09.012759 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.012637    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvgkt\""
Jul 28 16:03:09.013172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.013153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:03:09.013172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.013164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pxp\""
Jul 28 16:03:09.013172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.013176    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:03:09.088370 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.088325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:09.088514 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:09.088480    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:09.088566 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:09.088554    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:10.08853837 +0000 UTC m=+36.640136344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:09.151261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.151172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ph7dx" event={"ID":"d57eebac-dec7-4853-b9e0-420e03e1bd5c","Type":"ContainerStarted","Data":"16316d9c415dd60be2ae22da096af1474dfffaa586152947b741d7b8372dc8b0"}
Jul 28 16:03:09.153594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.153569    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="5c4ada347b7979f3236dab2bb2dd77f338626fbe96bb0140c46c87efc4c63edf" exitCode=0
Jul 28 16:03:09.153707 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:09.153603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"5c4ada347b7979f3236dab2bb2dd77f338626fbe96bb0140c46c87efc4c63edf"}
Jul 28 16:03:10.009615 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:10.009580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:10.012664 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:10.012642    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:03:10.095349 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:10.095314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:10.095558 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:10.095442    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:10.095558 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:10.095510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:12.095488189 +0000 UTC m=+38.647086159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:12.110356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:12.110316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:12.111142 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:12.110491    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:12.111142 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:12.110586    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:16.110564478 +0000 UTC m=+42.662162453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:15.171904 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:15.171867    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="48ffcb74c2c48d4a484b3cd6c44ae9651b322867cf137270f679ec8bdc5311f4" exitCode=0
Jul 28 16:03:15.172433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:15.171941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"48ffcb74c2c48d4a484b3cd6c44ae9651b322867cf137270f679ec8bdc5311f4"}
Jul 28 16:03:15.173377 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:15.173351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ph7dx" event={"ID":"d57eebac-dec7-4853-b9e0-420e03e1bd5c","Type":"ContainerStarted","Data":"383a7b9befecf9fc57f7088803658995600edeae38112b450ed9e04563add25a"}
Jul 28 16:03:15.214293 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:15.214245    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ph7dx" podStartSLOduration=1.636294843 podStartE2EDuration="7.21422886s" totalImagesPullingTime="5.577934017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:15.213877564 +0000 UTC m=+41.765475562" watchObservedRunningTime="2026-07-28 16:03:15.21422886 +0000 UTC m=+41.765826853"
Jul 28 16:03:16.139256 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:16.139210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:16.139429 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:16.139331    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:16.139429 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:16.139382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:24.139369103 +0000 UTC m=+50.690967074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:16.178594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:16.178564    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f" containerID="45762bde02b7e5098a7100ba964134550881e13bad87fd7406c60181946f1017" exitCode=0
Jul 28 16:03:16.178955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:16.178653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerDied","Data":"45762bde02b7e5098a7100ba964134550881e13bad87fd7406c60181946f1017"}
Jul 28 16:03:17.184155 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:17.184119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" event={"ID":"4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f","Type":"ContainerStarted","Data":"b97c2d04bd3a520cd781105072af4cd738f86217759e1ed12920fdedf9459d31"}
Jul 28 16:03:17.207154 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:17.207110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvgkp" podStartSLOduration=8.444757795 podStartE2EDuration="43.207093871s" totalImagesPullingTime="34.762336076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:17.206366026 +0000 UTC m=+43.757964023" watchObservedRunningTime="2026-07-28 16:03:17.207093871 +0000 UTC m=+43.758691863"
Jul 28 16:03:22.159784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.159757    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"]
Jul 28 16:03:22.161736 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.161719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.164394 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.164368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 16:03:22.165187 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.165164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 16:03:22.165263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.165234    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 16:03:22.165302 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.165269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 16:03:22.165518 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.165501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4gw4t\""
Jul 28 16:03:22.173442 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.173424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"]
Jul 28 16:03:22.277901 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.277864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48n67\" (UniqueName: \"kubernetes.io/projected/a304a7f1-175e-45e2-8d83-d2b7a791c434-kube-api-access-48n67\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.278070 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.277911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.278070 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.278020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a304a7f1-175e-45e2-8d83-d2b7a791c434-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.378450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.378418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n67\" (UniqueName: \"kubernetes.io/projected/a304a7f1-175e-45e2-8d83-d2b7a791c434-kube-api-access-48n67\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.378544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.378460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.378595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.378539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a304a7f1-175e-45e2-8d83-d2b7a791c434-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.378648 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:22.378630    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:22.378709 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:22.378699    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:22.878683469 +0000 UTC m=+49.430281444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:22.379298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.379279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a304a7f1-175e-45e2-8d83-d2b7a791c434-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.388336 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.388305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n67\" (UniqueName: \"kubernetes.io/projected/a304a7f1-175e-45e2-8d83-d2b7a791c434-kube-api-access-48n67\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.882256 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:22.882208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:22.882430 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:22.882367    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:22.882472 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:22.882435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:23.882419296 +0000 UTC m=+50.434017268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:23.888858 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:23.888822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:23.889246 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:23.888946    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:23.889246 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:23.889000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:25.888986238 +0000 UTC m=+52.440584209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:24.191405 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:24.191373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:24.191564 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:24.191545    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:24.191624 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:24.191615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:40.191598073 +0000 UTC m=+66.743196043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:25.903807 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:25.903750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:25.904226 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:25.903919    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:25.904226 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:25.903995    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:29.903977061 +0000 UTC m=+56.455575040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:27.539734 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.539700    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pfgp6"]
Jul 28 16:03:27.542328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.542313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.544811 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.544787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 16:03:27.544951 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.544790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbttq\""
Jul 28 16:03:27.545319 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.545303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 16:03:27.545403 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.545334    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 16:03:27.545466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.545338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 16:03:27.552938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.552919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfgp6"]
Jul 28 16:03:27.715184 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.715144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-data-volume\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.715184 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.715185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.715417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.715272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-crio-socket\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.715417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.715297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9hw\" (UniqueName: \"kubernetes.io/projected/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-api-access-vk9hw\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.715417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.715337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.815963 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.815875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816078 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:27.816042    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:27.816132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-data-volume\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816132 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:27.816120    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:28.316099297 +0000 UTC m=+54.867697269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:27.816245 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816245 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-crio-socket\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816341 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:27.816300    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:27.816341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9hw\" (UniqueName: \"kubernetes.io/projected/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-api-access-vk9hw\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816460 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-data-volume\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.816460 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:27.816355    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:28.316342953 +0000 UTC m=+54.867940924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:27.816460 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.816301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-crio-socket\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:27.824615 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:27.824594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9hw\" (UniqueName: \"kubernetes.io/projected/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-api-access-vk9hw\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:28.320404 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:28.320344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:28.320591 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:28.320448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:28.320591 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:28.320499    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:28.320591 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:28.320542    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:28.320591 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:28.320575    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:29.320557711 +0000 UTC m=+55.872155685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:28.320733 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:28.320597    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:29.320585407 +0000 UTC m=+55.872183378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:29.327915 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:29.327877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:29.328284 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.328021    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:29.328284 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:29.328053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:29.328284 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.328137    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:31.328118432 +0000 UTC m=+57.879716419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:29.328284 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.328172    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:29.328284 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.328226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:31.328215331 +0000 UTC m=+57.879813303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:29.511955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:29.511927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqjnl_03033695-3e53-43bc-b5e7-c121d4dd9fe0/dns-node-resolver/0.log"
Jul 28 16:03:29.931441 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:29.931391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:29.931593 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.931533    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:29.931647 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:29.931603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:37.931588829 +0000 UTC m=+64.483186804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:30.112565 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:30.112539    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c6pdm_86b6b215-9318-4e1d-b112-6fb7eb1e5025/node-ca/0.log"
Jul 28 16:03:31.113038 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:31.113013    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ph7dx_d57eebac-dec7-4853-b9e0-420e03e1bd5c/serve-healthcheck-canary/0.log"
Jul 28 16:03:31.344582 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:31.344543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:31.344747 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:31.344615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:31.344747 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:31.344686    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:31.344747 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:31.344697    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:31.344747 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:31.344748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:35.344731616 +0000 UTC m=+61.896329601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:31.344909 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:31.344763    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:35.344756646 +0000 UTC m=+61.896354616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:32.032470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.032439    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"]
Jul 28 16:03:32.062224 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.062198    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vbvkl"]
Jul 28 16:03:32.062367 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.062338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"
Jul 28 16:03:32.064437 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.064407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 16:03:32.064568 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.064407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ggj52\""
Jul 28 16:03:32.064568 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.064483    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:03:32.081385 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.081358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"]
Jul 28 16:03:32.081496 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.081392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vbvkl"]
Jul 28 16:03:32.081496 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.081471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.083666 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.083648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f7ssb\""
Jul 28 16:03:32.083781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.083690    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 16:03:32.083781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.083703    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:03:32.083781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.083736    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 16:03:32.083939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.083738    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 16:03:32.087918 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.087903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 16:03:32.130832 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.130804    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"]
Jul 28 16:03:32.151302 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.151275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"]
Jul 28 16:03:32.151440 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.151390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.154163 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.153939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 16:03:32.154595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.154367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-f29lj\""
Jul 28 16:03:32.155409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.155336    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 16:03:32.155641 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.155598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:03:32.155797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.155765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 16:03:32.171536 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.171514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"]
Jul 28 16:03:32.171536 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.171541    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"]
Jul 28 16:03:32.171657 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.171612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.173476 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.173459    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 16:03:32.173696 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.173680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 16:03:32.173806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.173693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:03:32.173806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.173709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 16:03:32.173806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.173739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4g2m9\""
Jul 28 16:03:32.251825 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.251793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwtc\" (UniqueName: \"kubernetes.io/projected/75e6180b-9257-4a8d-8b42-30aadcc3c76a-kube-api-access-lcwtc\") pod \"volume-data-source-validator-549c6d54cf-4t4j8\" (UID: \"75e6180b-9257-4a8d-8b42-30aadcc3c76a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"
Jul 28 16:03:32.251825 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.251824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h524k\" (UniqueName: \"kubernetes.io/projected/139a823e-a65d-4fd2-add0-ee066edd494b-kube-api-access-h524k\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.252043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.251914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.252043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.251945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/139a823e-a65d-4fd2-add0-ee066edd494b-serving-cert\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.252043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.251980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-config\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.252043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.252000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bdl\" (UniqueName: \"kubernetes.io/projected/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-kube-api-access-s4bdl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.252189 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.252044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.252189 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.252107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-trusted-ca\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.353470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7tn\" (UniqueName: \"kubernetes.io/projected/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-kube-api-access-nd7tn\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.353470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.353684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-trusted-ca\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.353684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwtc\" (UniqueName: \"kubernetes.io/projected/75e6180b-9257-4a8d-8b42-30aadcc3c76a-kube-api-access-lcwtc\") pod \"volume-data-source-validator-549c6d54cf-4t4j8\" (UID: \"75e6180b-9257-4a8d-8b42-30aadcc3c76a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"
Jul 28 16:03:32.353684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h524k\" (UniqueName: \"kubernetes.io/projected/139a823e-a65d-4fd2-add0-ee066edd494b-kube-api-access-h524k\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.353684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-config\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.353684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.353975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/139a823e-a65d-4fd2-add0-ee066edd494b-serving-cert\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.353975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-config\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.353975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.353975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.353787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bdl\" (UniqueName: \"kubernetes.io/projected/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-kube-api-access-s4bdl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.354427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.354402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.354582 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.354563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-trusted-ca\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.354647 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.354587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/139a823e-a65d-4fd2-add0-ee066edd494b-config\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.355886 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.355866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.355982 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.355941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/139a823e-a65d-4fd2-add0-ee066edd494b-serving-cert\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.363179 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.363148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h524k\" (UniqueName: \"kubernetes.io/projected/139a823e-a65d-4fd2-add0-ee066edd494b-kube-api-access-h524k\") pod \"console-operator-584dcb6b54-vbvkl\" (UID: \"139a823e-a65d-4fd2-add0-ee066edd494b\") " pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.363324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.363308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwtc\" (UniqueName: \"kubernetes.io/projected/75e6180b-9257-4a8d-8b42-30aadcc3c76a-kube-api-access-lcwtc\") pod \"volume-data-source-validator-549c6d54cf-4t4j8\" (UID: \"75e6180b-9257-4a8d-8b42-30aadcc3c76a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"
Jul 28 16:03:32.363426 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.363410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bdl\" (UniqueName: \"kubernetes.io/projected/f3be8fcf-eef5-4e7e-b55e-256eae8b17b6-kube-api-access-s4bdl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8\" (UID: \"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.371345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.371330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"
Jul 28 16:03:32.389647 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.389627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:32.454691 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.454647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.454832 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.454726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7tn\" (UniqueName: \"kubernetes.io/projected/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-kube-api-access-nd7tn\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.454924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.454842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-config\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.455468 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.455445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-config\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.456940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.456907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.462095 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.462069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"
Jul 28 16:03:32.465036 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.465003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7tn\" (UniqueName: \"kubernetes.io/projected/e28b7ae3-c40d-4caa-acee-0a1a72f33b87-kube-api-access-nd7tn\") pod \"service-ca-operator-5d6c75f84f-ms6wp\" (UID: \"e28b7ae3-c40d-4caa-acee-0a1a72f33b87\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.480442 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.479889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"
Jul 28 16:03:32.537083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.537019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8"]
Jul 28 16:03:32.544160 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:32.544127    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e6180b_9257_4a8d_8b42_30aadcc3c76a.slice/crio-812760d385ee655625a51035d294d29edc8ecfc5734ada66a9e412312dacb4c1 WatchSource:0}: Error finding container 812760d385ee655625a51035d294d29edc8ecfc5734ada66a9e412312dacb4c1: Status 404 returned error can't find the container with id 812760d385ee655625a51035d294d29edc8ecfc5734ada66a9e412312dacb4c1
Jul 28 16:03:32.560157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.560088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vbvkl"]
Jul 28 16:03:32.585787 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:32.585745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139a823e_a65d_4fd2_add0_ee066edd494b.slice/crio-f34053a5eb0e7122ed5bf051dfac3715d9ca1a8abf15c9a5c77aa1b4dd05d1bb WatchSource:0}: Error finding container f34053a5eb0e7122ed5bf051dfac3715d9ca1a8abf15c9a5c77aa1b4dd05d1bb: Status 404 returned error can't find the container with id f34053a5eb0e7122ed5bf051dfac3715d9ca1a8abf15c9a5c77aa1b4dd05d1bb
Jul 28 16:03:32.645427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.645396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8"]
Jul 28 16:03:32.648585 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:32.648557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3be8fcf_eef5_4e7e_b55e_256eae8b17b6.slice/crio-e8f6110b290da014452def7e8eb2ed96935a0f5405c34d563a926e8c72a5896b WatchSource:0}: Error finding container e8f6110b290da014452def7e8eb2ed96935a0f5405c34d563a926e8c72a5896b: Status 404 returned error can't find the container with id e8f6110b290da014452def7e8eb2ed96935a0f5405c34d563a926e8c72a5896b
Jul 28 16:03:32.663962 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:32.663935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp"]
Jul 28 16:03:32.667243 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:32.667219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28b7ae3_c40d_4caa_acee_0a1a72f33b87.slice/crio-83030ff1f9072a5f2eb1f59c89aee9bd7ee1489bebb6c18f65726a756a844387 WatchSource:0}: Error finding container 83030ff1f9072a5f2eb1f59c89aee9bd7ee1489bebb6c18f65726a756a844387: Status 404 returned error can't find the container with id 83030ff1f9072a5f2eb1f59c89aee9bd7ee1489bebb6c18f65726a756a844387
Jul 28 16:03:33.225081 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:33.225039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp" event={"ID":"e28b7ae3-c40d-4caa-acee-0a1a72f33b87","Type":"ContainerStarted","Data":"83030ff1f9072a5f2eb1f59c89aee9bd7ee1489bebb6c18f65726a756a844387"}
Jul 28 16:03:33.226605 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:33.226580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8" event={"ID":"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6","Type":"ContainerStarted","Data":"e8f6110b290da014452def7e8eb2ed96935a0f5405c34d563a926e8c72a5896b"}
Jul 28 16:03:33.228014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:33.227985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" event={"ID":"139a823e-a65d-4fd2-add0-ee066edd494b","Type":"ContainerStarted","Data":"f34053a5eb0e7122ed5bf051dfac3715d9ca1a8abf15c9a5c77aa1b4dd05d1bb"}
Jul 28 16:03:33.229168 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:33.229140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8" event={"ID":"75e6180b-9257-4a8d-8b42-30aadcc3c76a","Type":"ContainerStarted","Data":"812760d385ee655625a51035d294d29edc8ecfc5734ada66a9e412312dacb4c1"}
Jul 28 16:03:34.148232 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:34.148198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9lx"
Jul 28 16:03:34.234213 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:34.234129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8" event={"ID":"75e6180b-9257-4a8d-8b42-30aadcc3c76a","Type":"ContainerStarted","Data":"8006857a71fe218d487fa222c61568f5823e263db9d169c3c65ce198e122f17f"}
Jul 28 16:03:34.251379 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:34.251328    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-4t4j8" podStartSLOduration=0.83395673 podStartE2EDuration="2.251309462s" totalImagesPullingTime="1.417352732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:34.250494099 +0000 UTC m=+60.802092093" watchObservedRunningTime="2026-07-28 16:03:34.251309462 +0000 UTC m=+60.802907458"
Jul 28 16:03:35.380386 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:35.380358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:35.380763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:35.380443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:35.380763 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:35.380481    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 16:03:35.380763 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:35.380543    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:43.380524892 +0000 UTC m=+69.932122880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : configmap "kube-rbac-proxy" not found
Jul 28 16:03:35.380763 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:35.380556    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:35.380763 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:35.380599    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:43.38058849 +0000 UTC m=+69.932186475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:36.240867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.240821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp" event={"ID":"e28b7ae3-c40d-4caa-acee-0a1a72f33b87","Type":"ContainerStarted","Data":"7579dd23ca8ed0c465915b7b7279d50f636e1fd574a77985e23c18ea783f2feb"}
Jul 28 16:03:36.242511 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.242486    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/0.log"
Jul 28 16:03:36.242627 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.242530    2579 generic.go:356] "Generic (PLEG): container finished" podID="139a823e-a65d-4fd2-add0-ee066edd494b" containerID="ed0e4c7257d7072ade07581b226939e58ae54ec40048e414fd25f6cbc0bb0b9d" exitCode=255
Jul 28 16:03:36.242627 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.242562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" event={"ID":"139a823e-a65d-4fd2-add0-ee066edd494b","Type":"ContainerDied","Data":"ed0e4c7257d7072ade07581b226939e58ae54ec40048e414fd25f6cbc0bb0b9d"}
Jul 28 16:03:36.242781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.242768    2579 scope.go:117] "RemoveContainer" containerID="ed0e4c7257d7072ade07581b226939e58ae54ec40048e414fd25f6cbc0bb0b9d"
Jul 28 16:03:36.261684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:36.261632    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp" podStartSLOduration=1.5701059769999999 podStartE2EDuration="4.2616113s" totalImagesPullingTime="2.691505323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:36.260517916 +0000 UTC m=+62.812115910" watchObservedRunningTime="2026-07-28 16:03:36.2616113 +0000 UTC m=+62.813209296"
Jul 28 16:03:37.247423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.247396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8" event={"ID":"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6","Type":"ContainerStarted","Data":"bd1c17b9bc18c5c46d90e2dc4a8dd74a79395351259f952e93a39c82b720329b"}
Jul 28 16:03:37.249052 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249016    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:03:37.249563 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249534    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/0.log"
Jul 28 16:03:37.249656 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249593    2579 generic.go:356] "Generic (PLEG): container finished" podID="139a823e-a65d-4fd2-add0-ee066edd494b" containerID="2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef" exitCode=255
Jul 28 16:03:37.249713 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" event={"ID":"139a823e-a65d-4fd2-add0-ee066edd494b","Type":"ContainerDied","Data":"2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef"}
Jul 28 16:03:37.249751 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249738    2579 scope.go:117] "RemoveContainer" containerID="ed0e4c7257d7072ade07581b226939e58ae54ec40048e414fd25f6cbc0bb0b9d"
Jul 28 16:03:37.250083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.249933    2579 scope.go:117] "RemoveContainer" containerID="2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef"
Jul 28 16:03:37.250195 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:37.250142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vbvkl_openshift-console-operator(139a823e-a65d-4fd2-add0-ee066edd494b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" podUID="139a823e-a65d-4fd2-add0-ee066edd494b"
Jul 28 16:03:37.267836 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:37.267774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8" podStartSLOduration=0.732068971 podStartE2EDuration="5.267751903s" totalImagesPullingTime="4.535682932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:37.266800611 +0000 UTC m=+63.818398595" watchObservedRunningTime="2026-07-28 16:03:37.267751903 +0000 UTC m=+63.819349897"
Jul 28 16:03:38.000127 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:38.000086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:38.000306 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:38.000241    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:38.000355 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:38.000307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls podName:a304a7f1-175e-45e2-8d83-d2b7a791c434 nodeName:}" failed. No retries permitted until 2026-07-28 16:03:54.000291423 +0000 UTC m=+80.551889415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j7wxw" (UID: "a304a7f1-175e-45e2-8d83-d2b7a791c434") : secret "cluster-monitoring-operator-tls" not found
Jul 28 16:03:38.253636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:38.253553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:03:38.254028 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:38.253889    2579 scope.go:117] "RemoveContainer" containerID="2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef"
Jul 28 16:03:38.254065 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:38.254024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vbvkl_openshift-console-operator(139a823e-a65d-4fd2-add0-ee066edd494b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" podUID="139a823e-a65d-4fd2-add0-ee066edd494b"
Jul 28 16:03:39.611522 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.611465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:03:39.614651 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.614632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:03:39.622704 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:39.622684    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 16:03:39.622761 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:39.622740    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs podName:0e77f42d-6f86-4e08-b27e-ce9f7323662b nodeName:}" failed. No retries permitted until 2026-07-28 16:04:43.622725773 +0000 UTC m=+130.174323746 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs") pod "network-metrics-daemon-ztqjn" (UID: "0e77f42d-6f86-4e08-b27e-ce9f7323662b") : secret "metrics-daemon-secret" not found
Jul 28 16:03:39.712821 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.712780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:39.714980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.714962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:03:39.722022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.722001    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-fhknh"]
Jul 28 16:03:39.724669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.724647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.725209 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.725185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8109b844-aca8-4f61-b52c-61dc7132a484-original-pull-secret\") pod \"global-pull-secret-syncer-jmktt\" (UID: \"8109b844-aca8-4f61-b52c-61dc7132a484\") " pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:39.727321 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.727299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 16:03:39.727634 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.727343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 16:03:39.731265 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.731244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k87k6\""
Jul 28 16:03:39.731422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.731244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 16:03:39.731553 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.731380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 16:03:39.733267 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.733244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-fhknh"]
Jul 28 16:03:39.814153 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.814124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-cabundle\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.814322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.814163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-key\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.814322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.814206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmwx\" (UniqueName: \"kubernetes.io/projected/4d4c1185-f89d-4b1d-b606-0e315fce741c-kube-api-access-thmwx\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.814322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.814267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:39.816381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.816365    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:03:39.826615 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.826598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:03:39.837395 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.837371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8cj\" (UniqueName: \"kubernetes.io/projected/7caceb3f-a665-4ef7-bde8-d714d0f762fb-kube-api-access-rr8cj\") pod \"network-check-target-q8vsl\" (UID: \"7caceb3f-a665-4ef7-bde8-d714d0f762fb\") " pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:39.915088 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.914994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmwx\" (UniqueName: \"kubernetes.io/projected/4d4c1185-f89d-4b1d-b606-0e315fce741c-kube-api-access-thmwx\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.915088 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.915089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-cabundle\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.915251 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.915107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-key\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.915698 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.915676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-cabundle\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.917509 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.917487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d4c1185-f89d-4b1d-b606-0e315fce741c-signing-key\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.925211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.925192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmwx\" (UniqueName: \"kubernetes.io/projected/4d4c1185-f89d-4b1d-b606-0e315fce741c-kube-api-access-thmwx\") pod \"service-ca-658d7b7c78-fhknh\" (UID: \"4d4c1185-f89d-4b1d-b606-0e315fce741c\") " pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:39.926292 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.926277    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvgkt\""
Jul 28 16:03:39.934800 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:39.934785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:40.022051 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.022015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmktt"
Jul 28 16:03:40.039729 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.039699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-fhknh"
Jul 28 16:03:40.063794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.063684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8vsl"]
Jul 28 16:03:40.069824 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:40.069448    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caceb3f_a665_4ef7_bde8_d714d0f762fb.slice/crio-e89b55bd110de0ce8f7b01051509aecd73a3172fb0513544edb39da014fd0dc3 WatchSource:0}: Error finding container e89b55bd110de0ce8f7b01051509aecd73a3172fb0513544edb39da014fd0dc3: Status 404 returned error can't find the container with id e89b55bd110de0ce8f7b01051509aecd73a3172fb0513544edb39da014fd0dc3
Jul 28 16:03:40.153622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.153590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jmktt"]
Jul 28 16:03:40.156368 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:40.156340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8109b844_aca8_4f61_b52c_61dc7132a484.slice/crio-d7de9fcf6d65e872bf90c520a9fef243a4b9e3e86e7cd306fa7e9ef01c9f6d76 WatchSource:0}: Error finding container d7de9fcf6d65e872bf90c520a9fef243a4b9e3e86e7cd306fa7e9ef01c9f6d76: Status 404 returned error can't find the container with id d7de9fcf6d65e872bf90c520a9fef243a4b9e3e86e7cd306fa7e9ef01c9f6d76
Jul 28 16:03:40.173433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.173379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-fhknh"]
Jul 28 16:03:40.176410 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:40.176389    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4c1185_f89d_4b1d_b606_0e315fce741c.slice/crio-f4ccf06637f07e94b3950f7dd610276b60f7367f04fb95572970d9e7635a0f30 WatchSource:0}: Error finding container f4ccf06637f07e94b3950f7dd610276b60f7367f04fb95572970d9e7635a0f30: Status 404 returned error can't find the container with id f4ccf06637f07e94b3950f7dd610276b60f7367f04fb95572970d9e7635a0f30
Jul 28 16:03:40.217325 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.217303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:03:40.217421 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:40.217410    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 16:03:40.217477 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:40.217456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls podName:cb4d19b7-6b21-462a-9db2-11febdef81e4 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:12.217443149 +0000 UTC m=+98.769041121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls") pod "dns-default-5srrc" (UID: "cb4d19b7-6b21-462a-9db2-11febdef81e4") : secret "dns-default-metrics-tls" not found
Jul 28 16:03:40.260888 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.260832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8vsl" event={"ID":"7caceb3f-a665-4ef7-bde8-d714d0f762fb","Type":"ContainerStarted","Data":"e89b55bd110de0ce8f7b01051509aecd73a3172fb0513544edb39da014fd0dc3"}
Jul 28 16:03:40.262818 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.262774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-fhknh" event={"ID":"4d4c1185-f89d-4b1d-b606-0e315fce741c","Type":"ContainerStarted","Data":"c31555c2c57c1a6e5316cd97f28879de943f496c8e484ca6206123c063514b90"}
Jul 28 16:03:40.262972 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.262832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-fhknh" event={"ID":"4d4c1185-f89d-4b1d-b606-0e315fce741c","Type":"ContainerStarted","Data":"f4ccf06637f07e94b3950f7dd610276b60f7367f04fb95572970d9e7635a0f30"}
Jul 28 16:03:40.263954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.263928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jmktt" event={"ID":"8109b844-aca8-4f61-b52c-61dc7132a484","Type":"ContainerStarted","Data":"d7de9fcf6d65e872bf90c520a9fef243a4b9e3e86e7cd306fa7e9ef01c9f6d76"}
Jul 28 16:03:40.283612 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:40.283565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-fhknh" podStartSLOduration=1.283550679 podStartE2EDuration="1.283550679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:40.282672469 +0000 UTC m=+66.834270452" watchObservedRunningTime="2026-07-28 16:03:40.283550679 +0000 UTC m=+66.835148695"
Jul 28 16:03:42.390218 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:42.390058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:42.390218 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:42.390132    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:42.390879 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:42.390597    2579 scope.go:117] "RemoveContainer" containerID="2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef"
Jul 28 16:03:42.390879 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:42.390829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vbvkl_openshift-console-operator(139a823e-a65d-4fd2-add0-ee066edd494b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" podUID="139a823e-a65d-4fd2-add0-ee066edd494b"
Jul 28 16:03:43.445397 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:43.445349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:43.445795 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:43.445499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:43.445795 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:43.445647    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:43.445795 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:03:43.445722    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls podName:f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec nodeName:}" failed. No retries permitted until 2026-07-28 16:03:59.445702379 +0000 UTC m=+85.997300354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfgp6" (UID: "f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec") : secret "insights-runtime-extractor-tls" not found
Jul 28 16:03:43.445993 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:43.445956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:48.297158 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:48.297121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jmktt" event={"ID":"8109b844-aca8-4f61-b52c-61dc7132a484","Type":"ContainerStarted","Data":"ab89a0cd1f27f1615c0660a415e748d7f8bd16f97fa0b10e81c905dbb02cd730"}
Jul 28 16:03:48.298588 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:48.298565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8vsl" event={"ID":"7caceb3f-a665-4ef7-bde8-d714d0f762fb","Type":"ContainerStarted","Data":"53e9984770cf229592548fe0c31504abbdb5a18028606ff322b0d27d6a4a5e73"}
Jul 28 16:03:48.298689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:48.298648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:03:48.317899 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:48.317837    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jmktt" podStartSLOduration=67.215653245 podStartE2EDuration="1m14.317824164s" totalImagesPullingTime="7.102170919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:48.317486805 +0000 UTC m=+74.869084799" watchObservedRunningTime="2026-07-28 16:03:48.317824164 +0000 UTC m=+74.869422207"
Jul 28 16:03:48.338443 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:48.338408    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8vsl" podStartSLOduration=67.162995714 podStartE2EDuration="1m14.338395822s" totalImagesPullingTime="7.175400108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:48.338293828 +0000 UTC m=+74.889891820" watchObservedRunningTime="2026-07-28 16:03:48.338395822 +0000 UTC m=+74.889993809"
Jul 28 16:03:54.039017 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:54.038984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:54.041359 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:54.041333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a304a7f1-175e-45e2-8d83-d2b7a791c434-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j7wxw\" (UID: \"a304a7f1-175e-45e2-8d83-d2b7a791c434\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:54.272876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:54.272822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4gw4t\""
Jul 28 16:03:54.280581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:54.280557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"
Jul 28 16:03:54.398598 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:54.398570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw"]
Jul 28 16:03:54.401785 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:54.401757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda304a7f1_175e_45e2_8d83_d2b7a791c434.slice/crio-eeb9f5071419cd56b30edcafd85b88cc0c9733b61adac33637dd552e2e9deb8b WatchSource:0}: Error finding container eeb9f5071419cd56b30edcafd85b88cc0c9733b61adac33637dd552e2e9deb8b: Status 404 returned error can't find the container with id eeb9f5071419cd56b30edcafd85b88cc0c9733b61adac33637dd552e2e9deb8b
Jul 28 16:03:55.320839 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:55.320807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw" event={"ID":"a304a7f1-175e-45e2-8d83-d2b7a791c434","Type":"ContainerStarted","Data":"eeb9f5071419cd56b30edcafd85b88cc0c9733b61adac33637dd552e2e9deb8b"}
Jul 28 16:03:56.010274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:56.010246    2579 scope.go:117] "RemoveContainer" containerID="2262fdd7e02ba5d36f9741be98adc5b0847d4ef4f6cf8e79000b4674ae4b67ef"
Jul 28 16:03:57.329131 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.329105    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:03:57.329561 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.329184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" event={"ID":"139a823e-a65d-4fd2-add0-ee066edd494b","Type":"ContainerStarted","Data":"c4feef28fbe9d14d6fcad0590740c09c84dca7f2743dfe9d10841ba13bfdc3f8"}
Jul 28 16:03:57.329561 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.329498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:57.330794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.330771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw" event={"ID":"a304a7f1-175e-45e2-8d83-d2b7a791c434","Type":"ContainerStarted","Data":"4297a2ea4fb072a86a71ba262eae1e701c251fc71de54bf8d5dbc8f38fede4a2"}
Jul 28 16:03:57.334579 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.334562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl"
Jul 28 16:03:57.351108 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.351072    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-vbvkl" podStartSLOduration=22.582541137 podStartE2EDuration="25.351058874s" totalImagesPullingTime="2.768517737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:57.350716167 +0000 UTC m=+83.902314162" watchObservedRunningTime="2026-07-28 16:03:57.351058874 +0000 UTC m=+83.902656867"
Jul 28 16:03:57.368634 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:57.368599    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j7wxw" podStartSLOduration=33.331768181 podStartE2EDuration="35.368588724s" totalImagesPullingTime="2.036820543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:57.367972265 +0000 UTC m=+83.919570258" watchObservedRunningTime="2026-07-28 16:03:57.368588724 +0000 UTC m=+83.920186717"
Jul 28 16:03:59.479646 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:59.479596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:59.481999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:59.481976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfgp6\" (UID: \"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec\") " pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:59.653689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:59.653655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbttq\""
Jul 28 16:03:59.662240 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:59.662214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfgp6"
Jul 28 16:03:59.788890 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:03:59.788842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfgp6"]
Jul 28 16:03:59.792146 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:03:59.792122    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf691cc0e_5d31_44ad_a29a_ea1f5fe2c3ec.slice/crio-f04d2d076594e896d2f45cf13fbbfd706780179b702023569d908811541fac74 WatchSource:0}: Error finding container f04d2d076594e896d2f45cf13fbbfd706780179b702023569d908811541fac74: Status 404 returned error can't find the container with id f04d2d076594e896d2f45cf13fbbfd706780179b702023569d908811541fac74
Jul 28 16:04:00.345922 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:00.345881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfgp6" event={"ID":"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec","Type":"ContainerStarted","Data":"f04d2d076594e896d2f45cf13fbbfd706780179b702023569d908811541fac74"}
Jul 28 16:04:04.767029 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.766946    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"]
Jul 28 16:04:04.769038 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.769021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:04.771716 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.771652    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76988f478c-7mhcl"]
Jul 28 16:04:04.773454 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.773436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.777841 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:04.777808    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-142-59.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-142-59.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret"
Jul 28 16:04:04.777841 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:04.777811    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-142-59.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-142-59.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Jul 28 16:04:04.777841 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:04.777838    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-8qmb9\" is forbidden: User \"system:node:ip-10-0-142-59.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-142-59.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8qmb9\"" type="*v1.Secret"
Jul 28 16:04:04.778113 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:04.777877    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-142-59.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-142-59.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Jul 28 16:04:04.778113 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:04.777915    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-142-59.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-142-59.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Jul 28 16:04:04.778113 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.778047    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4vn2q\""
Jul 28 16:04:04.778579 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.778563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 16:04:04.778677 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.778570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 16:04:04.819544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.819513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"]
Jul 28 16:04:04.848774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.848745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76988f478c-7mhcl"]
Jul 28 16:04:04.889490 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.889451    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-8btsp"]
Jul 28 16:04:04.891707 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.891692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:04.895706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.895686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 16:04:04.895885 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.895866    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2glp2\""
Jul 28 16:04:04.895940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.895895    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 16:04:04.915009 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.914974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8btsp"]
Jul 28 16:04:04.924382 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlcdv\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924540 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924540 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924540 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:04.924540 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-installation-pull-secrets\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:04.924683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:04.924683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:04.924627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.021826 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.021746    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76988f478c-7mhcl"]
Jul 28 16:04:05.022015 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:05.021995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-rlcdv registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-76988f478c-7mhcl" podUID="fe5efae5-894b-4c0c-88aa-0c706917324d"
Jul 28 16:04:05.025554 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlcdv\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrnv\" (UniqueName: \"kubernetes.io/projected/405fca12-c251-41b9-92d7-41e987525982-kube-api-access-chrnv\") pod \"downloads-7c97dc4555-8btsp\" (UID: \"405fca12-c251-41b9-92d7-41e987525982\") " pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:05.025715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-installation-pull-secrets\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:05.025944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.025907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.026365 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.026339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.027460 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.027414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:05.027688 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.027600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.029533 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.029501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ed9dca5-b538-4613-a9d3-6da6a3a1e731-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tznhk\" (UID: \"4ed9dca5-b538-4613-a9d3-6da6a3a1e731\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:05.055212 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.055179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlcdv\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.064612 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.064549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.078364 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.078326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"
Jul 28 16:04:05.126722 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.126621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrnv\" (UniqueName: \"kubernetes.io/projected/405fca12-c251-41b9-92d7-41e987525982-kube-api-access-chrnv\") pod \"downloads-7c97dc4555-8btsp\" (UID: \"405fca12-c251-41b9-92d7-41e987525982\") " pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:05.156186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.156148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrnv\" (UniqueName: \"kubernetes.io/projected/405fca12-c251-41b9-92d7-41e987525982-kube-api-access-chrnv\") pod \"downloads-7c97dc4555-8btsp\" (UID: \"405fca12-c251-41b9-92d7-41e987525982\") " pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:05.200506 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.200470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:05.269208 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.269138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tznhk"]
Jul 28 16:04:05.271324 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:05.271298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed9dca5_b538_4613_a9d3_6da6a3a1e731.slice/crio-cfc9c9c93074a870e7caff14e714d31ca1a7811b2081f22576818e1d8de4d97e WatchSource:0}: Error finding container cfc9c9c93074a870e7caff14e714d31ca1a7811b2081f22576818e1d8de4d97e: Status 404 returned error can't find the container with id cfc9c9c93074a870e7caff14e714d31ca1a7811b2081f22576818e1d8de4d97e
Jul 28 16:04:05.366572 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.366531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfgp6" event={"ID":"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec","Type":"ContainerStarted","Data":"4f158db8f79db70454e79595bc2c386500996afc31ad7f0b6f5e3937750f24b1"}
Jul 28 16:04:05.368211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.368190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.368458 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.368441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk" event={"ID":"4ed9dca5-b538-4613-a9d3-6da6a3a1e731","Type":"ContainerStarted","Data":"cfc9c9c93074a870e7caff14e714d31ca1a7811b2081f22576818e1d8de4d97e"}
Jul 28 16:04:05.374806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.374778    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8btsp"]
Jul 28 16:04:05.376840 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.376591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.378754 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:05.378725    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405fca12_c251_41b9_92d7_41e987525982.slice/crio-13170a519078b5aa590ce63320bcb14fcfd72f48a5e3ad9f50b36b3eedec6d67 WatchSource:0}: Error finding container 13170a519078b5aa590ce63320bcb14fcfd72f48a5e3ad9f50b36b3eedec6d67: Status 404 returned error can't find the container with id 13170a519078b5aa590ce63320bcb14fcfd72f48a5e3ad9f50b36b3eedec6d67
Jul 28 16:04:05.530599 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.530527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.530750 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.530598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.530750 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.530630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlcdv\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.530750 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.530690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.531111 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.531079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:04:05.531111 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.531102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:05.533380 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.533351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:05.533487 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.533352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv" (OuterVolumeSpecName: "kube-api-access-rlcdv") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "kube-api-access-rlcdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:05.632284 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.632247    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-bound-sa-token\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.632284 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.632277    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-certificates\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.632284 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.632290    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlcdv\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-kube-api-access-rlcdv\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.632499 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.632302    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5efae5-894b-4c0c-88aa-0c706917324d-ca-trust-extracted\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.679551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.679519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 16:04:05.689904 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.689876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.715258 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.715225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 16:04:05.720621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.720595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.834801 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.834655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.834801 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.834756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:05.837970 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.837938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:05.839997 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.839960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:05.936350 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.936311    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-image-registry-private-configuration\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.936350 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.936350    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5efae5-894b-4c0c-88aa-0c706917324d-registry-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:05.965587 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.965555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 16:04:05.966706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.966680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca\") pod \"image-registry-76988f478c-7mhcl\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") " pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:05.995588 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.995551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:05.998411 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:05.998388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.001999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.000779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 16:04:06.001999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.001125    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 16:04:06.001999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.001347    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 16:04:06.001999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.001532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 16:04:06.001999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.001754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d4kl9\""
Jul 28 16:04:06.002329 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.002079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 16:04:06.022249 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.022189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:06.029783 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:06.027697    2579 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: failed to sync secret cache: timed out waiting for the condition
Jul 28 16:04:06.029783 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:06.027785    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-installation-pull-secrets podName:fe5efae5-894b-4c0c-88aa-0c706917324d nodeName:}" failed. No retries permitted until 2026-07-28 16:04:06.527761223 +0000 UTC m=+93.079359197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-installation-pull-secrets") pod "image-registry-76988f478c-7mhcl" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d") : failed to sync secret cache: timed out waiting for the condition
Jul 28 16:04:06.037371 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.037343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca\") pod \"fe5efae5-894b-4c0c-88aa-0c706917324d\" (UID: \"fe5efae5-894b-4c0c-88aa-0c706917324d\") "
Jul 28 16:04:06.038283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.038245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe5efae5-894b-4c0c-88aa-0c706917324d" (UID: "fe5efae5-894b-4c0c-88aa-0c706917324d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:06.046455 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.045402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8qmb9\""
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqq4\" (UniqueName: \"kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.138938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.138695    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5efae5-894b-4c0c-88aa-0c706917324d-trusted-ca\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:06.217684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.217467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 16:04:06.240202 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.240397 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.240464 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.240525 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.240576 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqq4\" (UniqueName: \"kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.240576 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.240549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.241117 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.241086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.242031 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.241962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.244882 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.244740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.245251 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.245222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.245348 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.245310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.255999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.255943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqq4\" (UniqueName: \"kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4\") pod \"console-5bb4d5977f-x7sr4\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") " pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.311994 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.311953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:06.375203 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.375161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76988f478c-7mhcl"
Jul 28 16:04:06.376368 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.376286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8btsp" event={"ID":"405fca12-c251-41b9-92d7-41e987525982","Type":"ContainerStarted","Data":"13170a519078b5aa590ce63320bcb14fcfd72f48a5e3ad9f50b36b3eedec6d67"}
Jul 28 16:04:06.423488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.423414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76988f478c-7mhcl"]
Jul 28 16:04:06.426957 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.426809    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76988f478c-7mhcl"]
Jul 28 16:04:06.542504 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:06.542464    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5efae5-894b-4c0c-88aa-0c706917324d-installation-pull-secrets\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:08.015226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:08.015191    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5efae5-894b-4c0c-88aa-0c706917324d" path="/var/lib/kubelet/pods/fe5efae5-894b-4c0c-88aa-0c706917324d/volumes"
Jul 28 16:04:09.215462 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:09.215394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:09.219984 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:09.219956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3c664b_ad87_4891_aed1_22f9184cd82b.slice/crio-a416286b42f7b471f38abb57bec01fab2c50c494101adc1a90b8b9c19ff25c6d WatchSource:0}: Error finding container a416286b42f7b471f38abb57bec01fab2c50c494101adc1a90b8b9c19ff25c6d: Status 404 returned error can't find the container with id a416286b42f7b471f38abb57bec01fab2c50c494101adc1a90b8b9c19ff25c6d
Jul 28 16:04:09.388013 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:09.387980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk" event={"ID":"4ed9dca5-b538-4613-a9d3-6da6a3a1e731","Type":"ContainerStarted","Data":"5d2fc92706ca605b3fa03bd580dd6303920aafd08458da2348f8b68c2135a1d2"}
Jul 28 16:04:09.389317 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:09.389281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4d5977f-x7sr4" event={"ID":"2f3c664b-ad87-4891-aed1-22f9184cd82b","Type":"ContainerStarted","Data":"a416286b42f7b471f38abb57bec01fab2c50c494101adc1a90b8b9c19ff25c6d"}
Jul 28 16:04:09.411630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:09.411581    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tznhk" podStartSLOduration=1.618888635 podStartE2EDuration="5.411565299s" totalImagesPullingTime="3.792676664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:09.41011603 +0000 UTC m=+95.961714022" watchObservedRunningTime="2026-07-28 16:04:09.411565299 +0000 UTC m=+95.963163350"
Jul 28 16:04:09.998250 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:09.998205    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:04:10.022867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.022817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.027777 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.027718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:04:10.040104 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.040070    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 16:04:10.161345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.161310    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"]
Jul 28 16:04:10.173972 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.173933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5hb\" (UniqueName: \"kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174304 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174304 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.174403 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.174319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.175173 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.175149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.179182 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.179001    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 16:04:10.180758 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.180550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 16:04:10.181199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.181179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dghdv\""
Jul 28 16:04:10.181311 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.181216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 16:04:10.182372 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.182348    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"]
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58014279-5258-496b-afd5-c6eda3e84e50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5hb\" (UniqueName: \"kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.275769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcmk\" (UniqueName: \"kubernetes.io/projected/58014279-5258-496b-afd5-c6eda3e84e50-kube-api-access-shcmk\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.276687 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.275996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.276687 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.276156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.276947 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.276899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.277308 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.277285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.282139 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.279591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.290081 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.290055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5hb\" (UniqueName: \"kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.291546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.291232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert\") pod \"console-97b8b55b-s4pjk\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") " pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.335034 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.335003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:10.377328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.376367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.377328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.376417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcmk\" (UniqueName: \"kubernetes.io/projected/58014279-5258-496b-afd5-c6eda3e84e50-kube-api-access-shcmk\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.377328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.376488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58014279-5258-496b-afd5-c6eda3e84e50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.377328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.376548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.377619 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:10.377341    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 16:04:10.377619 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:10.377425    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls podName:58014279-5258-496b-afd5-c6eda3e84e50 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:10.877405188 +0000 UTC m=+97.429003174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-jmvqq" (UID: "58014279-5258-496b-afd5-c6eda3e84e50") : secret "prometheus-operator-tls" not found
Jul 28 16:04:10.377724 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.377695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58014279-5258-496b-afd5-c6eda3e84e50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.380757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.380707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.396100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.396044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcmk\" (UniqueName: \"kubernetes.io/projected/58014279-5258-496b-afd5-c6eda3e84e50-kube-api-access-shcmk\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.398492 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.398408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfgp6" event={"ID":"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec","Type":"ContainerStarted","Data":"ed40c9004d9f4945a811d4425c196210c3da549d30b9d2fe18c4bf6d6a158f05"}
Jul 28 16:04:10.529384 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.529241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:04:10.533431 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:10.533400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50beb9c_e446_469c_9495_9fe909274f81.slice/crio-e7534da0dcfbffd1a2ba8e52384bb13741bcb4503bc37757ced4793f9db89990 WatchSource:0}: Error finding container e7534da0dcfbffd1a2ba8e52384bb13741bcb4503bc37757ced4793f9db89990: Status 404 returned error can't find the container with id e7534da0dcfbffd1a2ba8e52384bb13741bcb4503bc37757ced4793f9db89990
Jul 28 16:04:10.883532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.883439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:10.886644 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:10.886594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/58014279-5258-496b-afd5-c6eda3e84e50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-jmvqq\" (UID: \"58014279-5258-496b-afd5-c6eda3e84e50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:11.089756 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:11.089512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"
Jul 28 16:04:11.315991 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:11.315903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-jmvqq"]
Jul 28 16:04:11.320553 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:11.320517    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58014279_5258_496b_afd5_c6eda3e84e50.slice/crio-22b0296d687cca19018417754cb81af3dd1d977088b2c0891efd4457656febb3 WatchSource:0}: Error finding container 22b0296d687cca19018417754cb81af3dd1d977088b2c0891efd4457656febb3: Status 404 returned error can't find the container with id 22b0296d687cca19018417754cb81af3dd1d977088b2c0891efd4457656febb3
Jul 28 16:04:11.406106 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:11.406072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq" event={"ID":"58014279-5258-496b-afd5-c6eda3e84e50","Type":"ContainerStarted","Data":"22b0296d687cca19018417754cb81af3dd1d977088b2c0891efd4457656febb3"}
Jul 28 16:04:11.407490 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:11.407449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97b8b55b-s4pjk" event={"ID":"b50beb9c-e446-469c-9495-9fe909274f81","Type":"ContainerStarted","Data":"e7534da0dcfbffd1a2ba8e52384bb13741bcb4503bc37757ced4793f9db89990"}
Jul 28 16:04:12.297756 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:12.297717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:04:12.312404 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:12.312296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4d19b7-6b21-462a-9db2-11febdef81e4-metrics-tls\") pod \"dns-default-5srrc\" (UID: \"cb4d19b7-6b21-462a-9db2-11febdef81e4\") " pod="openshift-dns/dns-default-5srrc"
Jul 28 16:04:12.580714 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:12.580634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ssj24\""
Jul 28 16:04:12.588610 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:12.588555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5srrc"
Jul 28 16:04:15.992058 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:15.991925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5srrc"]
Jul 28 16:04:16.431794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.431763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfgp6" event={"ID":"f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec","Type":"ContainerStarted","Data":"54a144d0570595ce7bc2eaee9025ff0e152c0eacb59df4c947f723a8eef278db"}
Jul 28 16:04:16.433548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.433522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq" event={"ID":"58014279-5258-496b-afd5-c6eda3e84e50","Type":"ContainerStarted","Data":"59a2f3c4e9cdb38f479972579194c5dffe6b97c9dc53502274965ac1b723e45d"}
Jul 28 16:04:16.433666 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.433569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq" event={"ID":"58014279-5258-496b-afd5-c6eda3e84e50","Type":"ContainerStarted","Data":"e5cc42fa1358d370153d6cc91939d188b93b3b12f1a3f9d7324b2c80a7ae1490"}
Jul 28 16:04:16.435018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.434999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97b8b55b-s4pjk" event={"ID":"b50beb9c-e446-469c-9495-9fe909274f81","Type":"ContainerStarted","Data":"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"}
Jul 28 16:04:16.436426 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.436407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4d5977f-x7sr4" event={"ID":"2f3c664b-ad87-4891-aed1-22f9184cd82b","Type":"ContainerStarted","Data":"9752aac553a16e83793634711f679fa783bc2a04cec4e3d75c3e2d90cfbcdedd"}
Jul 28 16:04:16.437519 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.437496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5srrc" event={"ID":"cb4d19b7-6b21-462a-9db2-11febdef81e4","Type":"ContainerStarted","Data":"79118dfade0e8a16956ee37faa4d557ec32ef310b602f495fa404b33fd0235fe"}
Jul 28 16:04:16.468328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.468273    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pfgp6" podStartSLOduration=33.586686089 podStartE2EDuration="49.468253866s" totalImagesPullingTime="15.881567777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:03:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:16.465995697 +0000 UTC m=+103.017593689" watchObservedRunningTime="2026-07-28 16:04:16.468253866 +0000 UTC m=+103.019851862"
Jul 28 16:04:16.489133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.489085    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-jmvqq" podStartSLOduration=1.978016891 podStartE2EDuration="6.489064708s" totalImagesPullingTime="4.511047817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:16.487811554 +0000 UTC m=+103.039409546" watchObservedRunningTime="2026-07-28 16:04:16.489064708 +0000 UTC m=+103.040662702"
Jul 28 16:04:16.517547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.517491    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-97b8b55b-s4pjk" podStartSLOduration=2.218961812 podStartE2EDuration="7.517476038s" totalImagesPullingTime="5.298514226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:16.515614323 +0000 UTC m=+103.067212317" watchObservedRunningTime="2026-07-28 16:04:16.517476038 +0000 UTC m=+103.069074009"
Jul 28 16:04:16.539347 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:16.539300    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bb4d5977f-x7sr4" podStartSLOduration=4.927359522 podStartE2EDuration="11.53928121s" totalImagesPullingTime="6.611921688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:16.538919579 +0000 UTC m=+103.090517572" watchObservedRunningTime="2026-07-28 16:04:16.53928121 +0000 UTC m=+103.090879205"
Jul 28 16:04:18.698670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.698578    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"]
Jul 28 16:04:18.716509 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.716444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.725263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.724105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"]
Jul 28 16:04:18.725263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.724458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mp5q9\""
Jul 28 16:04:18.725263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.724655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 16:04:18.725263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.724837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 16:04:18.740873 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.740721    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vhtwm"]
Jul 28 16:04:18.766174 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.766150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.769881 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.769822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 16:04:18.770098 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.770081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5dx87\""
Jul 28 16:04:18.770270 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.770247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 16:04:18.770466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.770451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 16:04:18.861652 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.861621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.862025 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.862000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgfp\" (UniqueName: \"kubernetes.io/projected/2c88adca-ca36-4788-8369-161058283316-kube-api-access-kjgfp\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.862522 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.862500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c88adca-ca36-4788-8369-161058283316-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.862744 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.862726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.966636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.966466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgfp\" (UniqueName: \"kubernetes.io/projected/2c88adca-ca36-4788-8369-161058283316-kube-api-access-kjgfp\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.966636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.966524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-textfile\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.966636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.966551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-root\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-wtmp\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4w5\" (UniqueName: \"kubernetes.io/projected/54375cb1-27ea-4ab7-b107-68a425ab4db0-kube-api-access-2t4w5\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c88adca-ca36-4788-8369-161058283316-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-metrics-client-ca\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-sys\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.967370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-tls\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:18.967529    2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 16:04:18.967887 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:04:18.967589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls podName:2c88adca-ca36-4788-8369-161058283316 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:19.467570403 +0000 UTC m=+106.019168376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-hd9xj" (UID: "2c88adca-ca36-4788-8369-161058283316") : secret "openshift-state-metrics-tls" not found
Jul 28 16:04:18.968959 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.968931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c88adca-ca36-4788-8369-161058283316-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.993035 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.992963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:18.996867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:18.994086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgfp\" (UniqueName: \"kubernetes.io/projected/2c88adca-ca36-4788-8369-161058283316-kube-api-access-kjgfp\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-textfile\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-root\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-wtmp\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4w5\" (UniqueName: \"kubernetes.io/projected/54375cb1-27ea-4ab7-b107-68a425ab4db0-kube-api-access-2t4w5\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-metrics-client-ca\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.067865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.068190 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-sys\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.068190 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.067947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-tls\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.068516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-wtmp\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.068789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-textfile\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.069274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.069329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-root\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.069711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54375cb1-27ea-4ab7-b107-68a425ab4db0-metrics-client-ca\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.070808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-tls\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.070950 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.070893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54375cb1-27ea-4ab7-b107-68a425ab4db0-sys\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.074544 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.074504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54375cb1-27ea-4ab7-b107-68a425ab4db0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.086979 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.086954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4w5\" (UniqueName: \"kubernetes.io/projected/54375cb1-27ea-4ab7-b107-68a425ab4db0-kube-api-access-2t4w5\") pod \"node-exporter-vhtwm\" (UID: \"54375cb1-27ea-4ab7-b107-68a425ab4db0\") " pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.304345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.304264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8vsl"
Jul 28 16:04:19.383949 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.383918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhtwm"
Jul 28 16:04:19.471745 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.471712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:19.474315 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.474290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c88adca-ca36-4788-8369-161058283316-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hd9xj\" (UID: \"2c88adca-ca36-4788-8369-161058283316\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:19.648247 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.648167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"
Jul 28 16:04:19.823303 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.822967    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:04:19.841933 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.841903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.845395 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.845371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 16:04:19.845598 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.845371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 16:04:19.845925 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.845747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 16:04:19.845925 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.845764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 16:04:19.846132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.846112    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 16:04:19.846194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.846134    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 16:04:19.846194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.846134    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qm5vd\""
Jul 28 16:04:19.851690 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.850739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 16:04:19.851690 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.851131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 16:04:19.861107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.860902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 16:04:19.875403 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfnt\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.875909 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.876358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.876358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.875973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.876358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.876060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.881477 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.881450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:04:19.977307 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfnt\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.977971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.977799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.978169 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.978031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.981020 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.978565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.981405 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.981360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.981906 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.981820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.982513 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.982460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.983150 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.983100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.983377 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.983351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.983921 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.983875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.984546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.984283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.984546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.984513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.985119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.985055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.985520 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.985497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:19.994706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:19.994676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfnt\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt\") pod \"alertmanager-main-0\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:20.154837 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.154799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:04:20.335445 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.335347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:20.335445 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.335424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:20.340940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.340911    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:20.461048 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.461016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:04:20.513500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:20.513466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:23.090596 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.090556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c86664fdb-5pq5p"]
Jul 28 16:04:23.225059 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.225025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c86664fdb-5pq5p"]
Jul 28 16:04:23.225229 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.225135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9wdxc\""
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228260    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 16:04:23.228378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.228207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7p48acstor2l3\""
Jul 28 16:04:23.315025 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.314991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5a56f7-335f-4258-8505-d259e2b34c5f-audit-log\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/7e5a56f7-335f-4258-8505-d259e2b34c5f-kube-api-access-grzmd\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-metrics-server-audit-profiles\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-client-certs\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315196 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-client-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.315435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.315326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-tls\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416339 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-tls\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416339 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5a56f7-335f-4258-8505-d259e2b34c5f-audit-log\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/7e5a56f7-335f-4258-8505-d259e2b34c5f-kube-api-access-grzmd\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-metrics-server-audit-profiles\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-client-certs\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-client-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.416806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.416750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e5a56f7-335f-4258-8505-d259e2b34c5f-audit-log\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.417467 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.417422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.417696 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.417673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e5a56f7-335f-4258-8505-d259e2b34c5f-metrics-server-audit-profiles\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.419455 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.419430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-client-ca-bundle\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.419589 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.419564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-client-certs\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.420577 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.420555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e5a56f7-335f-4258-8505-d259e2b34c5f-secret-metrics-server-tls\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.429761 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.429717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/7e5a56f7-335f-4258-8505-d259e2b34c5f-kube-api-access-grzmd\") pod \"metrics-server-c86664fdb-5pq5p\" (UID: \"7e5a56f7-335f-4258-8505-d259e2b34c5f\") " pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.536564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.536527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:23.956040 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.956002    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56599ff8b9-68b27"]
Jul 28 16:04:23.968117 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.968090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:23.970567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.970539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 16:04:23.970689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.970611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 16:04:23.970689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.970673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 16:04:23.970689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.970683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 16:04:23.971232 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.970973    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g6q2p\""
Jul 28 16:04:23.971232 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.971079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 16:04:23.975835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.975812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56599ff8b9-68b27"]
Jul 28 16:04:23.976757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:23.976737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 16:04:24.021672 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-serving-certs-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.021672 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-metrics-client-ca\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.021871 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.021871 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.021871 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.021976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-federate-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.022018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.021985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.022018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.022009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx58b\" (UniqueName: \"kubernetes.io/projected/0999f069-f015-4c7a-b7e4-96cd466835ee-kube-api-access-bx58b\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.068435 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:24.068401    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54375cb1_27ea_4ab7_b107_68a425ab4db0.slice/crio-cffe0e483f5f8572134fa7554ddcd2f6f3a8ec7a686d01ba7085860d75bb2c7d WatchSource:0}: Error finding container cffe0e483f5f8572134fa7554ddcd2f6f3a8ec7a686d01ba7085860d75bb2c7d: Status 404 returned error can't find the container with id cffe0e483f5f8572134fa7554ddcd2f6f3a8ec7a686d01ba7085860d75bb2c7d
Jul 28 16:04:24.125733 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.125697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-federate-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx58b\" (UniqueName: \"kubernetes.io/projected/0999f069-f015-4c7a-b7e4-96cd466835ee-kube-api-access-bx58b\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-serving-certs-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.133531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-metrics-client-ca\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.134621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.134374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-metrics-client-ca\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.139194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.139111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-serving-certs-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.140121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.140096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.151269 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.151214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-federate-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.153701 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.153652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-telemeter-client-tls\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.154676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.154633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.154791 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.154703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0999f069-f015-4c7a-b7e4-96cd466835ee-secret-telemeter-client\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.156692 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.156657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx58b\" (UniqueName: \"kubernetes.io/projected/0999f069-f015-4c7a-b7e4-96cd466835ee-kube-api-access-bx58b\") pod \"telemeter-client-56599ff8b9-68b27\" (UID: \"0999f069-f015-4c7a-b7e4-96cd466835ee\") " pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.222912 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.222806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj"]
Jul 28 16:04:24.241017 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.240992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c86664fdb-5pq5p"]
Jul 28 16:04:24.263416 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.263391    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:04:24.281223 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.281199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27"
Jul 28 16:04:24.297030 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:24.296968    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c88adca_ca36_4788_8369_161058283316.slice/crio-9a0a19c3c33af61221040d1762f228bd67add154292489d60ba03643da5081d2 WatchSource:0}: Error finding container 9a0a19c3c33af61221040d1762f228bd67add154292489d60ba03643da5081d2: Status 404 returned error can't find the container with id 9a0a19c3c33af61221040d1762f228bd67add154292489d60ba03643da5081d2
Jul 28 16:04:24.297717 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:24.297692    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5a56f7_335f_4258_8505_d259e2b34c5f.slice/crio-96b16a542d1a45bbf1616d58adb7ee9e3241a54b69cc490ca8d13d85a83a6eec WatchSource:0}: Error finding container 96b16a542d1a45bbf1616d58adb7ee9e3241a54b69cc490ca8d13d85a83a6eec: Status 404 returned error can't find the container with id 96b16a542d1a45bbf1616d58adb7ee9e3241a54b69cc490ca8d13d85a83a6eec
Jul 28 16:04:24.298804 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:24.298779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a9d44d_b04b_4687_b063_7e144493bd47.slice/crio-e057d4ee76c362f94e93229337e8d7a25d2120b0689972b4131cb4602f99e988 WatchSource:0}: Error finding container e057d4ee76c362f94e93229337e8d7a25d2120b0689972b4131cb4602f99e988: Status 404 returned error can't find the container with id e057d4ee76c362f94e93229337e8d7a25d2120b0689972b4131cb4602f99e988
Jul 28 16:04:24.471097 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.471060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhtwm" event={"ID":"54375cb1-27ea-4ab7-b107-68a425ab4db0","Type":"ContainerStarted","Data":"cffe0e483f5f8572134fa7554ddcd2f6f3a8ec7a686d01ba7085860d75bb2c7d"}
Jul 28 16:04:24.472386 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.472351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p" event={"ID":"7e5a56f7-335f-4258-8505-d259e2b34c5f","Type":"ContainerStarted","Data":"96b16a542d1a45bbf1616d58adb7ee9e3241a54b69cc490ca8d13d85a83a6eec"}
Jul 28 16:04:24.473604 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.473532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj" event={"ID":"2c88adca-ca36-4788-8369-161058283316","Type":"ContainerStarted","Data":"9a0a19c3c33af61221040d1762f228bd67add154292489d60ba03643da5081d2"}
Jul 28 16:04:24.474872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.474828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"e057d4ee76c362f94e93229337e8d7a25d2120b0689972b4131cb4602f99e988"}
Jul 28 16:04:24.871003 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.868660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:04:24.875638 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.875609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.884602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.884863    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.885066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.885293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.885506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d84jq\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.885795    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.886007    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.886192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 16:04:24.886903 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.886485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 16:04:24.887393 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.887191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.887600    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.893243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.893556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.893775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.893993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.894005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-61jb58hvbt96j\""
Jul 28 16:04:24.894330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.894183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph59g\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.946053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.947022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.947022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.947022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:24.947022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:24.945802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.003623 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.003537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56599ff8b9-68b27"]
Jul 28 16:04:25.007639 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:25.006348    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0999f069_f015_4c7a_b7e4_96cd466835ee.slice/crio-583a9a989311bf57b008540ea3af10005a6c30ebf02da04a41e758bb07700fa1 WatchSource:0}: Error finding container 583a9a989311bf57b008540ea3af10005a6c30ebf02da04a41e758bb07700fa1: Status 404 returned error can't find the container with id 583a9a989311bf57b008540ea3af10005a6c30ebf02da04a41e758bb07700fa1
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.049721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph59g\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.050659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.050659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.050659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.048988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.050880 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.050740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.053552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.052127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.053552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.052283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.053552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.052678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.055535 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.055190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.057597 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.057390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.058090 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.058029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.058877 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.058788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.059874 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.059806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.062963 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.062675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.063528 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.063216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.063810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.063790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.064524 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.064202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.065563 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.065106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph59g\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.065563 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.065509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.067555 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.067512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.068378 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.068354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.070941 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.070388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.070941 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.070708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.070941 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.070781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.272799 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.272766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:25.484548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.484436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27" event={"ID":"0999f069-f015-4c7a-b7e4-96cd466835ee","Type":"ContainerStarted","Data":"583a9a989311bf57b008540ea3af10005a6c30ebf02da04a41e758bb07700fa1"}
Jul 28 16:04:25.486147 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.486055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:04:25.489535 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:25.488408    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b77cd3_f35d_4bd5_9c12_3298ca2e0590.slice/crio-3a9268a92034cc40d1b694d87db6c28469747e909c25dc68f06518e11258753f WatchSource:0}: Error finding container 3a9268a92034cc40d1b694d87db6c28469747e909c25dc68f06518e11258753f: Status 404 returned error can't find the container with id 3a9268a92034cc40d1b694d87db6c28469747e909c25dc68f06518e11258753f
Jul 28 16:04:25.493584 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.490911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8btsp" event={"ID":"405fca12-c251-41b9-92d7-41e987525982","Type":"ContainerStarted","Data":"1628869fe41a2fee6746fc79bb74ed87e16e886daa1eed40a3d4697dc613899a"}
Jul 28 16:04:25.493584 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.491825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:25.496584 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.496537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-8btsp"
Jul 28 16:04:25.499703 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.499670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj" event={"ID":"2c88adca-ca36-4788-8369-161058283316","Type":"ContainerStarted","Data":"9a836ddf0d9aa23fad14d0ccfacbe674b0813d9c3f77520c8010b193e2e813fc"}
Jul 28 16:04:25.499803 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.499729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj" event={"ID":"2c88adca-ca36-4788-8369-161058283316","Type":"ContainerStarted","Data":"32da67601e4f3f4d6037f41a28bcc5f91ede75105d61feadb25d3ce615abbd1e"}
Jul 28 16:04:25.503409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.502611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5srrc" event={"ID":"cb4d19b7-6b21-462a-9db2-11febdef81e4","Type":"ContainerStarted","Data":"9d07a67f0f829312274b06f1cb48a6949316fbe7e61afc862e778fcbcb7badb6"}
Jul 28 16:04:25.503409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.502662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5srrc" event={"ID":"cb4d19b7-6b21-462a-9db2-11febdef81e4","Type":"ContainerStarted","Data":"1eeb35b9b3beb528dbe1f09894f42812cbec9bdf4cf948e62f345a3e86d87001"}
Jul 28 16:04:25.503409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.503347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5srrc"
Jul 28 16:04:25.511784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.511730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-8btsp" podStartSLOduration=2.077781069 podStartE2EDuration="21.511711645s" totalImagesPullingTime="19.433930576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:25.509421323 +0000 UTC m=+112.061019318" watchObservedRunningTime="2026-07-28 16:04:25.511711645 +0000 UTC m=+112.063309634"
Jul 28 16:04:25.559690 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:25.555520    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5srrc" podStartSLOduration=68.684124087 podStartE2EDuration="1m17.555488529s" totalImagesPullingTime="8.871364442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:25.531635875 +0000 UTC m=+112.083233859" watchObservedRunningTime="2026-07-28 16:04:25.555488529 +0000 UTC m=+112.107086524"
Jul 28 16:04:26.313300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:26.313142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:26.509663 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:26.509568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"3a9268a92034cc40d1b694d87db6c28469747e909c25dc68f06518e11258753f"}
Jul 28 16:04:29.531291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.530224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj" event={"ID":"2c88adca-ca36-4788-8369-161058283316","Type":"ContainerStarted","Data":"f4609003114516042c46eda039711856985d5cb52fbfe4b97e93af0e670358a1"}
Jul 28 16:04:29.532679 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.532654    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a" exitCode=0
Jul 28 16:04:29.532983 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.532801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"}
Jul 28 16:04:29.536161 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.536118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p" event={"ID":"7e5a56f7-335f-4258-8505-d259e2b34c5f","Type":"ContainerStarted","Data":"c8a96e9610996cc6ef2e7bdf1ef5d69a1856c166c734a6bbbebdf5f2ae5f6adc"}
Jul 28 16:04:29.538205 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.538140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27" event={"ID":"0999f069-f015-4c7a-b7e4-96cd466835ee","Type":"ContainerStarted","Data":"0491d98083e4d63766d72ada96b69d624960794b2b995ae0e89438b8a3e100de"}
Jul 28 16:04:29.567987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.567936    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hd9xj" podStartSLOduration=7.467293048 podStartE2EDuration="11.567916776s" totalImagesPullingTime="4.100623728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:29.55110346 +0000 UTC m=+116.102701454" watchObservedRunningTime="2026-07-28 16:04:29.567916776 +0000 UTC m=+116.119514769"
Jul 28 16:04:29.569250 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:29.569211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p" podStartSLOduration=1.874587729 podStartE2EDuration="6.569197588s" totalImagesPullingTime="4.694609859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:29.567004527 +0000 UTC m=+116.118602522" watchObservedRunningTime="2026-07-28 16:04:29.569197588 +0000 UTC m=+116.120795581"
Jul 28 16:04:30.545709 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.545668    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4" exitCode=0
Jul 28 16:04:30.546202 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.545766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"}
Jul 28 16:04:30.550406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.550377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27" event={"ID":"0999f069-f015-4c7a-b7e4-96cd466835ee","Type":"ContainerStarted","Data":"a193381b51d6833c708ff9bccde5349ba1b0ddffc6e506fdb5947817454b89a0"}
Jul 28 16:04:30.550605 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.550445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27" event={"ID":"0999f069-f015-4c7a-b7e4-96cd466835ee","Type":"ContainerStarted","Data":"13fccbb75c31ac7a8c14067d9721114d9271feacc46263772d787bb50125fcd5"}
Jul 28 16:04:30.554276 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.554250    2579 generic.go:356] "Generic (PLEG): container finished" podID="54375cb1-27ea-4ab7-b107-68a425ab4db0" containerID="230d9bcdb4a23154a3c6aa8c17b4d215a6f7f436bae2bf207158a58bde181c99" exitCode=0
Jul 28 16:04:30.554448 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.554418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhtwm" event={"ID":"54375cb1-27ea-4ab7-b107-68a425ab4db0","Type":"ContainerDied","Data":"230d9bcdb4a23154a3c6aa8c17b4d215a6f7f436bae2bf207158a58bde181c99"}
Jul 28 16:04:30.608503 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:30.608443    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56599ff8b9-68b27" podStartSLOduration=3.492064517 podStartE2EDuration="7.608418844s" totalImagesPullingTime="4.116354327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:30.605449182 +0000 UTC m=+117.157047178" watchObservedRunningTime="2026-07-28 16:04:30.608418844 +0000 UTC m=+117.160016839"
Jul 28 16:04:31.563053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:31.563012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhtwm" event={"ID":"54375cb1-27ea-4ab7-b107-68a425ab4db0","Type":"ContainerStarted","Data":"99effa505aab7e85051e2f120b32d447c17e8b92219cc3541bedc02418cbfc74"}
Jul 28 16:04:31.563504 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:31.563107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhtwm" event={"ID":"54375cb1-27ea-4ab7-b107-68a425ab4db0","Type":"ContainerStarted","Data":"00f29f0809fdfd55bd28faa03059d266e653066a9e5de7315a4ed7a1315a1661"}
Jul 28 16:04:31.587282 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:31.587211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vhtwm" podStartSLOduration=8.18231577 podStartE2EDuration="13.58718897s" totalImagesPullingTime="5.271300842s" totalInitContainerRuntime="133.572358ms" isStatefulPod=true podCreationTimestamp="2026-07-28 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:31.585409617 +0000 UTC m=+118.137007612" watchObservedRunningTime="2026-07-28 16:04:31.58718897 +0000 UTC m=+118.138786964"
Jul 28 16:04:32.570865 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:32.570800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"}
Jul 28 16:04:32.571295 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:32.570924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"}
Jul 28 16:04:32.571295 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:32.570971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"}
Jul 28 16:04:32.571295 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:32.571012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"}
Jul 28 16:04:33.583195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:33.583158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"}
Jul 28 16:04:35.596733 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:35.596691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerStarted","Data":"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"}
Jul 28 16:04:35.600113 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:35.600083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"}
Jul 28 16:04:35.624874 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:35.624746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.790597039 podStartE2EDuration="16.624721773s" totalImagesPullingTime="9.807348408s" totalInitContainerRuntime="26.776326ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:35.622716787 +0000 UTC m=+122.174314782" watchObservedRunningTime="2026-07-28 16:04:35.624721773 +0000 UTC m=+122.176319768"
Jul 28 16:04:36.518422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:36.518389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5srrc"
Jul 28 16:04:36.607217 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:36.607181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"}
Jul 28 16:04:38.619015 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:38.618978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"}
Jul 28 16:04:38.619306 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:38.619053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"}
Jul 28 16:04:38.619306 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:38.619081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"}
Jul 28 16:04:39.628079 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:39.628040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerStarted","Data":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"}
Jul 28 16:04:39.666299 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:39.666238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.416425785 podStartE2EDuration="15.666215904s" totalImagesPullingTime="11.230159862s" totalInitContainerRuntime="19.630257ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:39.664896719 +0000 UTC m=+126.216494714" watchObservedRunningTime="2026-07-28 16:04:39.666215904 +0000 UTC m=+126.217813898"
Jul 28 16:04:40.275232 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:40.275193    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:04:42.053350 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:42.053318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:04:43.536771 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.536737    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:43.537161 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.537034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:04:43.670837 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.670799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:04:43.673053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.673036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e77f42d-6f86-4e08-b27e-ce9f7323662b-metrics-certs\") pod \"network-metrics-daemon-ztqjn\" (UID: \"0e77f42d-6f86-4e08-b27e-ce9f7323662b\") " pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:04:43.822322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.822247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7pxp\""
Jul 28 16:04:43.830820 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:43.830798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztqjn"
Jul 28 16:04:44.020291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:44.020242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ztqjn"]
Jul 28 16:04:44.024468 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:04:44.024443    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e77f42d_6f86_4e08_b27e_ce9f7323662b.slice/crio-018bf5f35f836945106df01724ad7313939343210d30f33ac1d18d8893dc8e46 WatchSource:0}: Error finding container 018bf5f35f836945106df01724ad7313939343210d30f33ac1d18d8893dc8e46: Status 404 returned error can't find the container with id 018bf5f35f836945106df01724ad7313939343210d30f33ac1d18d8893dc8e46
Jul 28 16:04:44.655495 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:44.655458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztqjn" event={"ID":"0e77f42d-6f86-4e08-b27e-ce9f7323662b","Type":"ContainerStarted","Data":"018bf5f35f836945106df01724ad7313939343210d30f33ac1d18d8893dc8e46"}
Jul 28 16:04:45.536226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.536182    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bb4d5977f-x7sr4" podUID="2f3c664b-ad87-4891-aed1-22f9184cd82b" containerName="console" containerID="cri-o://9752aac553a16e83793634711f679fa783bc2a04cec4e3d75c3e2d90cfbcdedd" gracePeriod=15
Jul 28 16:04:45.661111 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.661077    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f3c664b-ad87-4891-aed1-22f9184cd82b" containerID="9752aac553a16e83793634711f679fa783bc2a04cec4e3d75c3e2d90cfbcdedd" exitCode=0
Jul 28 16:04:45.661445 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.661156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4d5977f-x7sr4" event={"ID":"2f3c664b-ad87-4891-aed1-22f9184cd82b","Type":"ContainerDied","Data":"9752aac553a16e83793634711f679fa783bc2a04cec4e3d75c3e2d90cfbcdedd"}
Jul 28 16:04:45.829432 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.829402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:45.892027 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.891991    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892027 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892253 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892253 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892253 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892253 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzqq4\" (UniqueName: \"kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4\") pod \"2f3c664b-ad87-4891-aed1-22f9184cd82b\" (UID: \"2f3c664b-ad87-4891-aed1-22f9184cd82b\") "
Jul 28 16:04:45.892520 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config" (OuterVolumeSpecName: "console-config") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:45.892589 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:45.892589 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.892572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:45.894369 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.894341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:45.894492 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.894397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:45.894492 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.894453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4" (OuterVolumeSpecName: "kube-api-access-qzqq4") pod "2f3c664b-ad87-4891-aed1-22f9184cd82b" (UID: "2f3c664b-ad87-4891-aed1-22f9184cd82b"). InnerVolumeSpecName "kube-api-access-qzqq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:45.993215 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993182    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-oauth-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.993215 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993213    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.993215 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993224    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-service-ca\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.993412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993232    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3c664b-ad87-4891-aed1-22f9184cd82b-console-serving-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.993412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993242    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f3c664b-ad87-4891-aed1-22f9184cd82b-oauth-serving-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.993412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:45.993250    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqq4\" (UniqueName: \"kubernetes.io/projected/2f3c664b-ad87-4891-aed1-22f9184cd82b-kube-api-access-qzqq4\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:04:46.666955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.666929    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb4d5977f-x7sr4"
Jul 28 16:04:46.667436 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.666954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb4d5977f-x7sr4" event={"ID":"2f3c664b-ad87-4891-aed1-22f9184cd82b","Type":"ContainerDied","Data":"a416286b42f7b471f38abb57bec01fab2c50c494101adc1a90b8b9c19ff25c6d"}
Jul 28 16:04:46.667436 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.667016    2579 scope.go:117] "RemoveContainer" containerID="9752aac553a16e83793634711f679fa783bc2a04cec4e3d75c3e2d90cfbcdedd"
Jul 28 16:04:46.668591 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.668567    2579 generic.go:356] "Generic (PLEG): container finished" podID="e28b7ae3-c40d-4caa-acee-0a1a72f33b87" containerID="7579dd23ca8ed0c465915b7b7279d50f636e1fd574a77985e23c18ea783f2feb" exitCode=0
Jul 28 16:04:46.668705 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.668629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp" event={"ID":"e28b7ae3-c40d-4caa-acee-0a1a72f33b87","Type":"ContainerDied","Data":"7579dd23ca8ed0c465915b7b7279d50f636e1fd574a77985e23c18ea783f2feb"}
Jul 28 16:04:46.668945 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.668930    2579 scope.go:117] "RemoveContainer" containerID="7579dd23ca8ed0c465915b7b7279d50f636e1fd574a77985e23c18ea783f2feb"
Jul 28 16:04:46.720235 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.720209    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:46.727096 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:46.727068    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bb4d5977f-x7sr4"]
Jul 28 16:04:47.674816 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:47.674769    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3be8fcf-eef5-4e7e-b55e-256eae8b17b6" containerID="bd1c17b9bc18c5c46d90e2dc4a8dd74a79395351259f952e93a39c82b720329b" exitCode=0
Jul 28 16:04:47.675247 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:47.674840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8" event={"ID":"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6","Type":"ContainerDied","Data":"bd1c17b9bc18c5c46d90e2dc4a8dd74a79395351259f952e93a39c82b720329b"}
Jul 28 16:04:47.675330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:47.675268    2579 scope.go:117] "RemoveContainer" containerID="bd1c17b9bc18c5c46d90e2dc4a8dd74a79395351259f952e93a39c82b720329b"
Jul 28 16:04:47.677843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:47.677822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ms6wp" event={"ID":"e28b7ae3-c40d-4caa-acee-0a1a72f33b87","Type":"ContainerStarted","Data":"a7bbc76f186b085626be722df2f7517a013ccc266848f0939e4f5a33b6a244cb"}
Jul 28 16:04:48.015367 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:48.015332    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3c664b-ad87-4891-aed1-22f9184cd82b" path="/var/lib/kubelet/pods/2f3c664b-ad87-4891-aed1-22f9184cd82b/volumes"
Jul 28 16:04:48.684160 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:48.684125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8" event={"ID":"f3be8fcf-eef5-4e7e-b55e-256eae8b17b6","Type":"ContainerStarted","Data":"c55deac8cd9fa625e1d65a8f925470978887ee840759288ed032389d51c91813"}
Jul 28 16:04:51.697994 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:51.697955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztqjn" event={"ID":"0e77f42d-6f86-4e08-b27e-ce9f7323662b","Type":"ContainerStarted","Data":"384da7bbdebbc3b4f04cdac9ec475cfb5d6a9bce997a769eaae04c51c78aee56"}
Jul 28 16:04:51.698457 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:51.698018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztqjn" event={"ID":"0e77f42d-6f86-4e08-b27e-ce9f7323662b","Type":"ContainerStarted","Data":"d216514c87c742157174a10ebdd10f5cb511edf0de3f6bb6ef978d8e6a5c96cd"}
Jul 28 16:04:51.718028 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:04:51.717984    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ztqjn" podStartSLOduration=131.106082242 podStartE2EDuration="2m17.717969051s" totalImagesPullingTime="6.611886809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:51.715188893 +0000 UTC m=+138.266786886" watchObservedRunningTime="2026-07-28 16:04:51.717969051 +0000 UTC m=+138.269567044"
Jul 28 16:05:03.542091 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:03.542059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:05:03.545788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:03.545759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c86664fdb-5pq5p"
Jul 28 16:05:07.072769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.072705    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-97b8b55b-s4pjk" podUID="b50beb9c-e446-469c-9495-9fe909274f81" containerName="console" containerID="cri-o://4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f" gracePeriod=15
Jul 28 16:05:07.369457 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.369427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:05:07.484334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484521 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484521 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484521 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484521 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484735 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484735 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c5hb\" (UniqueName: \"kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb\") pod \"b50beb9c-e446-469c-9495-9fe909274f81\" (UID: \"b50beb9c-e446-469c-9495-9fe909274f81\") "
Jul 28 16:05:07.484892 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config" (OuterVolumeSpecName: "console-config") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:07.484978 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484902    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca" (OuterVolumeSpecName: "service-ca") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:07.484978 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:07.484978 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.484944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:07.485093 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.485033    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-oauth-serving-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.485093 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.485045    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-console-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.485093 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.485055    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-trusted-ca-bundle\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.485093 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.485064    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50beb9c-e446-469c-9495-9fe909274f81-service-ca\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.486509 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.486480    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:07.486606 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.486509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:07.486669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.486635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb" (OuterVolumeSpecName: "kube-api-access-5c5hb") pod "b50beb9c-e446-469c-9495-9fe909274f81" (UID: "b50beb9c-e446-469c-9495-9fe909274f81"). InnerVolumeSpecName "kube-api-access-5c5hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:07.585835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.585748    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-oauth-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.585835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.585789    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5hb\" (UniqueName: \"kubernetes.io/projected/b50beb9c-e446-469c-9495-9fe909274f81-kube-api-access-5c5hb\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.585835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.585809    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50beb9c-e446-469c-9495-9fe909274f81-console-serving-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:07.766931 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.766891    2579 generic.go:356] "Generic (PLEG): container finished" podID="b50beb9c-e446-469c-9495-9fe909274f81" containerID="4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f" exitCode=0
Jul 28 16:05:07.766931 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.766936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97b8b55b-s4pjk"
Jul 28 16:05:07.767125 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.766934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97b8b55b-s4pjk" event={"ID":"b50beb9c-e446-469c-9495-9fe909274f81","Type":"ContainerDied","Data":"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"}
Jul 28 16:05:07.767125 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.767068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97b8b55b-s4pjk" event={"ID":"b50beb9c-e446-469c-9495-9fe909274f81","Type":"ContainerDied","Data":"e7534da0dcfbffd1a2ba8e52384bb13741bcb4503bc37757ced4793f9db89990"}
Jul 28 16:05:07.767125 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.767099    2579 scope.go:117] "RemoveContainer" containerID="4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"
Jul 28 16:05:07.776578 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.776561    2579 scope.go:117] "RemoveContainer" containerID="4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"
Jul 28 16:05:07.776837 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:07.776818    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f\": container with ID starting with 4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f not found: ID does not exist" containerID="4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"
Jul 28 16:05:07.776914 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.776868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f"} err="failed to get container status \"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f\": rpc error: code = NotFound desc = could not find container \"4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f\": container with ID starting with 4785d3dccb2ad6ea0e6f3157477bd76afa5bc51910843151136ae1e77a81b07f not found: ID does not exist"
Jul 28 16:05:07.802597 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.802569    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:05:07.808801 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:07.808775    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-97b8b55b-s4pjk"]
Jul 28 16:05:08.014799 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:08.014765    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50beb9c-e446-469c-9495-9fe909274f81" path="/var/lib/kubelet/pods/b50beb9c-e446-469c-9495-9fe909274f81/volumes"
Jul 28 16:05:25.275091 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:25.275037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:25.294424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:25.294394    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:25.863447 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:25.863415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:39.087971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.087932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:05:39.088414 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088241    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="alertmanager" containerID="cri-o://8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69" gracePeriod=120
Jul 28 16:05:39.088414 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088304    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-metric" containerID="cri-o://a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6" gracePeriod=120
Jul 28 16:05:39.088414 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088353    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="prom-label-proxy" containerID="cri-o://c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669" gracePeriod=120
Jul 28 16:05:39.088414 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088397    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="config-reloader" containerID="cri-o://436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214" gracePeriod=120
Jul 28 16:05:39.088625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088399    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy" containerID="cri-o://d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26" gracePeriod=120
Jul 28 16:05:39.088625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.088333    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-web" containerID="cri-o://962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e" gracePeriod=120
Jul 28 16:05:39.907274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907244    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669" exitCode=0
Jul 28 16:05:39.907274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907267    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26" exitCode=0
Jul 28 16:05:39.907274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907273    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214" exitCode=0
Jul 28 16:05:39.907274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907278    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69" exitCode=0
Jul 28 16:05:39.907557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"}
Jul 28 16:05:39.907557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"}
Jul 28 16:05:39.907557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"}
Jul 28 16:05:39.907557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:39.907407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"}
Jul 28 16:05:40.335079 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.335055    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:05:40.378597 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378567    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.378772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.378772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.378772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.378772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbfnt\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.378772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.378994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.379143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.379028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy\") pod \"35a9d44d-b04b-4687-b063-7e144493bd47\" (UID: \"35a9d44d-b04b-4687-b063-7e144493bd47\") "
Jul 28 16:05:40.383022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.382632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:40.383022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.382899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:40.385497 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.385455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:40.385656 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.385506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt" (OuterVolumeSpecName: "kube-api-access-nbfnt") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "kube-api-access-nbfnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:40.387030 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.386977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.390538 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.390509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:40.391655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.391626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume" (OuterVolumeSpecName: "config-volume") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.393440 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.393417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.394117 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.394093    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.394224 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.394128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out" (OuterVolumeSpecName: "config-out") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:40.394895 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.394831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.396358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.396326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.403516 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.403495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config" (OuterVolumeSpecName: "web-config") pod "35a9d44d-b04b-4687-b063-7e144493bd47" (UID: "35a9d44d-b04b-4687-b063-7e144493bd47"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.480157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480110    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-metrics-client-ca\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480156    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-config-volume\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480168    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480179    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480188    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-tls-assets\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480197    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480206    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480238    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-web-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480247    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfnt\" (UniqueName: \"kubernetes.io/projected/35a9d44d-b04b-4687-b063-7e144493bd47-kube-api-access-nbfnt\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480255    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-cluster-tls-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480264    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a9d44d-b04b-4687-b063-7e144493bd47-secret-alertmanager-main-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480272    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-config-out\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.480449 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.480280    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a9d44d-b04b-4687-b063-7e144493bd47-alertmanager-main-db\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.914412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914326    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6" exitCode=0
Jul 28 16:05:40.914412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914358    2579 generic.go:356] "Generic (PLEG): container finished" podID="35a9d44d-b04b-4687-b063-7e144493bd47" containerID="962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e" exitCode=0
Jul 28 16:05:40.914595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"}
Jul 28 16:05:40.914595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:05:40.914595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"}
Jul 28 16:05:40.914595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a9d44d-b04b-4687-b063-7e144493bd47","Type":"ContainerDied","Data":"e057d4ee76c362f94e93229337e8d7a25d2120b0689972b4131cb4602f99e988"}
Jul 28 16:05:40.914736 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.914580    2579 scope.go:117] "RemoveContainer" containerID="c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"
Jul 28 16:05:40.923144 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.923127    2579 scope.go:117] "RemoveContainer" containerID="a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"
Jul 28 16:05:40.930069 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.930053    2579 scope.go:117] "RemoveContainer" containerID="d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"
Jul 28 16:05:40.936466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.936449    2579 scope.go:117] "RemoveContainer" containerID="962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"
Jul 28 16:05:40.940625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.940602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:05:40.943408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.943386    2579 scope.go:117] "RemoveContainer" containerID="436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"
Jul 28 16:05:40.948119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.948101    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:05:40.950692 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.950679    2579 scope.go:117] "RemoveContainer" containerID="8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"
Jul 28 16:05:40.957415 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.957396    2579 scope.go:117] "RemoveContainer" containerID="e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"
Jul 28 16:05:40.964026 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964010    2579 scope.go:117] "RemoveContainer" containerID="c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"
Jul 28 16:05:40.964265 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.964248    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669\": container with ID starting with c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669 not found: ID does not exist" containerID="c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"
Jul 28 16:05:40.964324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964273    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"} err="failed to get container status \"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669\": rpc error: code = NotFound desc = could not find container \"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669\": container with ID starting with c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669 not found: ID does not exist"
Jul 28 16:05:40.964324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964291    2579 scope.go:117] "RemoveContainer" containerID="a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"
Jul 28 16:05:40.964538 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.964521    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6\": container with ID starting with a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6 not found: ID does not exist" containerID="a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"
Jul 28 16:05:40.964599 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964547    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"} err="failed to get container status \"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6\": rpc error: code = NotFound desc = could not find container \"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6\": container with ID starting with a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6 not found: ID does not exist"
Jul 28 16:05:40.964599 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964569    2579 scope.go:117] "RemoveContainer" containerID="d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"
Jul 28 16:05:40.964809 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.964790    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26\": container with ID starting with d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26 not found: ID does not exist" containerID="d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"
Jul 28 16:05:40.964867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964815    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"} err="failed to get container status \"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26\": rpc error: code = NotFound desc = could not find container \"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26\": container with ID starting with d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26 not found: ID does not exist"
Jul 28 16:05:40.964867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.964830    2579 scope.go:117] "RemoveContainer" containerID="962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"
Jul 28 16:05:40.965070 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.965054    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e\": container with ID starting with 962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e not found: ID does not exist" containerID="962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"
Jul 28 16:05:40.965138 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965078    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"} err="failed to get container status \"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e\": rpc error: code = NotFound desc = could not find container \"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e\": container with ID starting with 962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e not found: ID does not exist"
Jul 28 16:05:40.965138 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965097    2579 scope.go:117] "RemoveContainer" containerID="436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"
Jul 28 16:05:40.965327 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.965309    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214\": container with ID starting with 436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214 not found: ID does not exist" containerID="436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"
Jul 28 16:05:40.965362 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965335    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"} err="failed to get container status \"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214\": rpc error: code = NotFound desc = could not find container \"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214\": container with ID starting with 436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214 not found: ID does not exist"
Jul 28 16:05:40.965362 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965351    2579 scope.go:117] "RemoveContainer" containerID="8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"
Jul 28 16:05:40.965555 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.965540    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69\": container with ID starting with 8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69 not found: ID does not exist" containerID="8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"
Jul 28 16:05:40.965595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965558    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"} err="failed to get container status \"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69\": rpc error: code = NotFound desc = could not find container \"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69\": container with ID starting with 8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69 not found: ID does not exist"
Jul 28 16:05:40.965595 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965572    2579 scope.go:117] "RemoveContainer" containerID="e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"
Jul 28 16:05:40.965765 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:40.965750    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a\": container with ID starting with e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a not found: ID does not exist" containerID="e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"
Jul 28 16:05:40.965810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"} err="failed to get container status \"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a\": rpc error: code = NotFound desc = could not find container \"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a\": container with ID starting with e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a not found: ID does not exist"
Jul 28 16:05:40.965810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965786    2579 scope.go:117] "RemoveContainer" containerID="c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"
Jul 28 16:05:40.965989 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669"} err="failed to get container status \"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669\": rpc error: code = NotFound desc = could not find container \"c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669\": container with ID starting with c3f7af114e73cc9641cf485f2dff02fb18741032f7d9c18a2c2822603c4f3669 not found: ID does not exist"
Jul 28 16:05:40.966030 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.965990    2579 scope.go:117] "RemoveContainer" containerID="a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"
Jul 28 16:05:40.966147 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966131    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6"} err="failed to get container status \"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6\": rpc error: code = NotFound desc = could not find container \"a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6\": container with ID starting with a37914449d98229a012806ab9893ff2a576ca1bc0352d7723010a3444b83a5f6 not found: ID does not exist"
Jul 28 16:05:40.966186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966147    2579 scope.go:117] "RemoveContainer" containerID="d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"
Jul 28 16:05:40.966329 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26"} err="failed to get container status \"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26\": rpc error: code = NotFound desc = could not find container \"d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26\": container with ID starting with d79c0d846739e682daf266aa753436ec0cb3d2dc088282a93a9ae40a2271bd26 not found: ID does not exist"
Jul 28 16:05:40.966373 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966330    2579 scope.go:117] "RemoveContainer" containerID="962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"
Jul 28 16:05:40.966506 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966491    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e"} err="failed to get container status \"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e\": rpc error: code = NotFound desc = could not find container \"962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e\": container with ID starting with 962befda954c574544ee946862905e6f6ee7488a818dbd9b68f8e3fe58c6180e not found: ID does not exist"
Jul 28 16:05:40.966547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966507    2579 scope.go:117] "RemoveContainer" containerID="436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"
Jul 28 16:05:40.966687 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214"} err="failed to get container status \"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214\": rpc error: code = NotFound desc = could not find container \"436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214\": container with ID starting with 436d76409651da85a315cba95669d8d43ad489e3feffd078774797b6d0645214 not found: ID does not exist"
Jul 28 16:05:40.966727 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966688    2579 scope.go:117] "RemoveContainer" containerID="8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"
Jul 28 16:05:40.966875 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966835    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69"} err="failed to get container status \"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69\": rpc error: code = NotFound desc = could not find container \"8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69\": container with ID starting with 8509f573789a06325c1e034affd9abf5eb3ae19ce71c4e74183d5f9551862f69 not found: ID does not exist"
Jul 28 16:05:40.966875 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.966865    2579 scope.go:117] "RemoveContainer" containerID="e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"
Jul 28 16:05:40.967045 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:40.967028    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a"} err="failed to get container status \"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a\": rpc error: code = NotFound desc = could not find container \"e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a\": container with ID starting with e3c8be7c69573e1a2bf4fcbc731df01da13bcc6e1c30c26cc7c5591ae2cf430a not found: ID does not exist"
Jul 28 16:05:42.015224 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:42.015188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" path="/var/lib/kubelet/pods/35a9d44d-b04b-4687-b063-7e144493bd47/volumes"
Jul 28 16:05:43.429597 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429556    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429959    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="prometheus" containerID="cri-o://be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd" gracePeriod=600
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429998    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045" gracePeriod=600
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429996    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="thanos-sidecar" containerID="cri-o://dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a" gracePeriod=600
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.430057    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-web" containerID="cri-o://22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6" gracePeriod=600
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429967    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy" containerID="cri-o://5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f" gracePeriod=600
Jul 28 16:05:43.430107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.429999    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="config-reloader" containerID="cri-o://ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786" gracePeriod=600
Jul 28 16:05:43.679744 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.679679    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:43.811299 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811299 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph59g\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.811843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811929    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811944    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle\") pod \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\" (UID: \"23b77cd3-f35d-4bd5-9c12-3298ca2e0590\") "
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.811966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812246    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812267    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812282    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.812413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812296    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-metrics-client-ca\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.812876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.812876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.812669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:43.814027 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.813994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.814027 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.814013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.814704 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.814383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.814704 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.814415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.814968 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.814933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.815084 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.815033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g" (OuterVolumeSpecName: "kube-api-access-ph59g") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "kube-api-access-ph59g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:43.815374 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.815348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.815880 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.815840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.815955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.815901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:43.816362 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.816345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:43.816611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.816589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out" (OuterVolumeSpecName: "config-out") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:43.816786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.816767    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.817080 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.817063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config" (OuterVolumeSpecName: "config") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.825121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.825094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config" (OuterVolumeSpecName: "web-config") pod "23b77cd3-f35d-4bd5-9c12-3298ca2e0590" (UID: "23b77cd3-f35d-4bd5-9c12-3298ca2e0590"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:43.912669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912642    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912664    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-web-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912675    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912687    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-metrics-client-certs\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912700    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph59g\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-kube-api-access-ph59g\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912709    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912719    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912728    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-grpc-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912739    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config-out\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912751    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-tls-assets\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912763    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912778    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912792    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912806    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912819    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-secret-kube-rbac-proxy\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.912940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.912832    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23b77cd3-f35d-4bd5-9c12-3298ca2e0590-prometheus-k8s-db\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:05:43.929032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.928994    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045" exitCode=0
Jul 28 16:05:43.929032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929022    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f" exitCode=0
Jul 28 16:05:43.929032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929028    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6" exitCode=0
Jul 28 16:05:43.929032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929034    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a" exitCode=0
Jul 28 16:05:43.929032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929039    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786" exitCode=0
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929045    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd" exitCode=0
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929072    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"}
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"}
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"}
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"}
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929220    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.929272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"}
Jul 28 16:05:43.929552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"}
Jul 28 16:05:43.929552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.929333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23b77cd3-f35d-4bd5-9c12-3298ca2e0590","Type":"ContainerDied","Data":"3a9268a92034cc40d1b694d87db6c28469747e909c25dc68f06518e11258753f"}
Jul 28 16:05:43.937028 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.936897    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.943794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.943774    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.950663 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.950646    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.954788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.954766    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:43.957523 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.957501    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.960801 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.960783    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:43.964486 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.964470    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.971507 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.971491    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.978032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978015    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.978275 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.978255    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.978346 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978280    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.978346 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978302    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.978536 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.978522    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.978584 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978538    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.978584 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978555    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.978789 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.978773    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.978832 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978792    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.978832 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.978804    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.979068 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.979050    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.979112 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979070    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.979112 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979081    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.979283 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.979269    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.979332 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979285    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.979332 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979297    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.979536 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.979517    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.979581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979540    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.979581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979557    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.979769 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:05:43.979756    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.979809 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.979809 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.979782    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.980022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980004    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.980022 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980022    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.980227 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980208    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.980227 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980226    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.980441 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980421    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.980517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980443    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.980713 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.980783 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980715    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.981000 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.980975    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.981000 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981001    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.981263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981246    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.981317 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981265    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.981500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.981545 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981502    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.981732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981716    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.981732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981731    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.981917 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.981969 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.981917    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.982123 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982108    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.982167 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982123    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.982348 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982329    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.982416 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982349    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.982581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.982629 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982583    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.982808 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.982880 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982809    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.983011 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.982995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.983055 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983010    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.983221 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983203    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.983261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983222    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.983446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.983489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983447    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.983634 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983614    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.983685 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983634    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.983823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983808    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.983886 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.983822    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.984073 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984056    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.984130 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984075    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.984283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.984283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984275    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.984451 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984436    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.984500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984452    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.984622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984607    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.984622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984622    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.984784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.984784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984775    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.984977 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.985035 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.984977    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.985191 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985177    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.985278 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985191    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.985402 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985384    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.985461 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985404    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.985613 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985596    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.985659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985614    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.985796 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985781    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.985840 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985796    2579 scope.go:117] "RemoveContainer" containerID="5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"
Jul 28 16:05:43.986000 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985985    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045"} err="failed to get container status \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": rpc error: code = NotFound desc = could not find container \"5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045\": container with ID starting with 5e9feb95cdbf127bc6e2611d11c679b0a3018d8b38fa59b44ec3ae0fdb470045 not found: ID does not exist"
Jul 28 16:05:43.986000 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.985999    2579 scope.go:117] "RemoveContainer" containerID="5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"
Jul 28 16:05:43.986200 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f"} err="failed to get container status \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": rpc error: code = NotFound desc = could not find container \"5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f\": container with ID starting with 5aa99e75ecb9f58b7e9fba3d7a56f1ea1f2a6b2564d6dbb8cd34c78218bf1b2f not found: ID does not exist"
Jul 28 16:05:43.986242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986202    2579 scope.go:117] "RemoveContainer" containerID="22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"
Jul 28 16:05:43.986408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986390    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6"} err="failed to get container status \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": rpc error: code = NotFound desc = could not find container \"22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6\": container with ID starting with 22b96406065d62a06836fe877d0f60212ce912289d5483992c73d66ac5d775c6 not found: ID does not exist"
Jul 28 16:05:43.986478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986410    2579 scope.go:117] "RemoveContainer" containerID="dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"
Jul 28 16:05:43.986621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986605    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a"} err="failed to get container status \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": rpc error: code = NotFound desc = could not find container \"dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a\": container with ID starting with dd63381ec133df3122b2c0379f2370382a39bc53a08755c5cd71b65a179d4f9a not found: ID does not exist"
Jul 28 16:05:43.986667 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986622    2579 scope.go:117] "RemoveContainer" containerID="ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"
Jul 28 16:05:43.986802 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986785    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786"} err="failed to get container status \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": rpc error: code = NotFound desc = could not find container \"ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786\": container with ID starting with ccba63ec9b9a5906f7efecf44442f68b6a66e2fa5b622f64823c684ee9edd786 not found: ID does not exist"
Jul 28 16:05:43.986885 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.986804    2579 scope.go:117] "RemoveContainer" containerID="be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"
Jul 28 16:05:43.987030 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.987011    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd"} err="failed to get container status \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": rpc error: code = NotFound desc = could not find container \"be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd\": container with ID starting with be68daa62fe9fc5523d57ce92813da846c36144cbf91649384cd50cd4fdcb0cd not found: ID does not exist"
Jul 28 16:05:43.987030 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.987027    2579 scope.go:117] "RemoveContainer" containerID="6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"
Jul 28 16:05:43.987227 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.987209    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4"} err="failed to get container status \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": rpc error: code = NotFound desc = could not find container \"6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4\": container with ID starting with 6bb5ceb97b06543645135b9687b3ddff1040ab850df74bb645e20434091968e4 not found: ID does not exist"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.993674    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="prom-label-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994338    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="prom-label-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994359    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="thanos-sidecar"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="thanos-sidecar"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994390    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994400    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994408    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994428    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994436    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994460    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="init-config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="init-config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994479    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-metric"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-metric"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994505    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="prometheus"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994514    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="prometheus"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994526    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="alertmanager"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994533    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="alertmanager"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994548    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994556    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994575    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994582    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994600    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994608    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994617    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-thanos"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-thanos"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994643    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="init-config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="init-config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994674    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3c664b-ad87-4891-aed1-22f9184cd82b" containerName="console"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994682    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3c664b-ad87-4891-aed1-22f9184cd82b" containerName="console"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994694    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50beb9c-e446-469c-9495-9fe909274f81" containerName="console"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994707    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50beb9c-e446-469c-9495-9fe909274f81" containerName="console"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994841    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="prom-label-proxy"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994868    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="config-reloader"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994878    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="alertmanager"
Jul 28 16:05:43.997763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994894    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3c664b-ad87-4891-aed1-22f9184cd82b" containerName="console"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994902    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="thanos-sidecar"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994909    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994922    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994931    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994938    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="kube-rbac-proxy-thanos"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994952    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-web"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994962    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="config-reloader"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9d44d-b04b-4687-b063-7e144493bd47" containerName="kube-rbac-proxy-metric"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" containerName="prometheus"
Jul 28 16:05:43.999072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:43.994991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50beb9c-e446-469c-9495-9fe909274f81" containerName="console"
Jul 28 16:05:44.000466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.000440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.003041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.002991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 16:05:44.003041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 16:05:44.003194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.002993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d84jq\""
Jul 28 16:05:44.003194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 16:05:44.003341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 16:05:44.003453 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-61jb58hvbt96j\""
Jul 28 16:05:44.003514 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 16:05:44.003566 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 16:05:44.003640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 16:05:44.003640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 16:05:44.003820 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 16:05:44.003945 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 16:05:44.003945 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.003871    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 16:05:44.004230 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.004206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 16:05:44.008460 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.008437    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 16:05:44.012077 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.012058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 16:05:44.014456 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.014432    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b77cd3-f35d-4bd5-9c12-3298ca2e0590" path="/var/lib/kubelet/pods/23b77cd3-f35d-4bd5-9c12-3298ca2e0590/volumes"
Jul 28 16:05:44.018107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.018085    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:44.113867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.113820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114009 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.113875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114009 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.113935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114009 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.113978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lkh\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-kube-api-access-s7lkh\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114136 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114136 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114136 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114136 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114309 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114309 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114309 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114309 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.114510 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.114488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.215995 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.215868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.216863 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.216613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.217817 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.217480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.218949 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.218925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.218964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219478 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lkh\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-kube-api-access-s7lkh\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219912 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.219978 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.219917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.220335 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.220312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.220409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.220361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.220409 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.220379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.221287 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.221268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.221572 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.221551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/185512b9-c6ad-499f-828e-50bc27899e4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.221639 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.221561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/185512b9-c6ad-499f-828e-50bc27899e4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.223073 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.223056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/185512b9-c6ad-499f-828e-50bc27899e4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.228381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.228356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lkh\" (UniqueName: \"kubernetes.io/projected/185512b9-c6ad-499f-828e-50bc27899e4c-kube-api-access-s7lkh\") pod \"prometheus-k8s-0\" (UID: \"185512b9-c6ad-499f-828e-50bc27899e4c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.310797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.310766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:05:44.450555 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.450413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:05:44.452922 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:05:44.452898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185512b9_c6ad_499f_828e_50bc27899e4c.slice/crio-2a00a715250b56f6f45b096de14f4e2e457de44201145adfca5d26f7052a90db WatchSource:0}: Error finding container 2a00a715250b56f6f45b096de14f4e2e457de44201145adfca5d26f7052a90db: Status 404 returned error can't find the container with id 2a00a715250b56f6f45b096de14f4e2e457de44201145adfca5d26f7052a90db
Jul 28 16:05:44.935212 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.935180    2579 generic.go:356] "Generic (PLEG): container finished" podID="185512b9-c6ad-499f-828e-50bc27899e4c" containerID="27a19e7fbe5adbdfa49fe3a3255b69c55eb6185aa740a034ba4ca3a950627117" exitCode=0
Jul 28 16:05:44.935434 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.935267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerDied","Data":"27a19e7fbe5adbdfa49fe3a3255b69c55eb6185aa740a034ba4ca3a950627117"}
Jul 28 16:05:44.935434 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:44.935345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"2a00a715250b56f6f45b096de14f4e2e457de44201145adfca5d26f7052a90db"}
Jul 28 16:05:45.942888 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.942837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"4ea396a86a016069feb7643923c395ee5138da6e855e87b47aa497d9ca32f667"}
Jul 28 16:05:45.943261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.942920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"dd2293d2259a20136772b49123ca369e4e9d7df67f3565d35fe26a27774b8d6a"}
Jul 28 16:05:45.943261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.942945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"48d8afae85d7bc0494014696a90e425a1d90141a2bf4d0a4f02224be2fcbbbd5"}
Jul 28 16:05:45.943261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.942979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"8acc95e35cca1cb907e45666fb5e6605963087586ba167bec767f4b46060ca8f"}
Jul 28 16:05:45.943261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.943004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"88ad7e3c7d4216cd95d807a7b7bc2c01e0ac1cced9e0c0487ae04a247df2c116"}
Jul 28 16:05:45.943261 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.943031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"185512b9-c6ad-499f-828e-50bc27899e4c","Type":"ContainerStarted","Data":"05c583212c18c00f21d458d4955bea74178781e67f3dd2e04a9fcdb0653de731"}
Jul 28 16:05:45.974343 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:45.974288    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.95522213 podStartE2EDuration="2.974270168s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.048038ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:45.974008867 +0000 UTC m=+192.525606862" watchObservedRunningTime="2026-07-28 16:05:45.974270168 +0000 UTC m=+192.525868163"
Jul 28 16:05:49.311863 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:05:49.311806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:06:44.311735 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:06:44.311697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:06:44.327756 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:06:44.327731    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:06:45.194662 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:06:45.194633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:07:33.819285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:07:33.819259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:07:33.821333 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:07:33.821311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:07:33.829326 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:07:33.829307    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 16:09:43.299861 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.299825    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77c6cf54b8-tbn6f"]
Jul 28 16:09:43.302558 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.302526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.304545 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.304522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 16:09:43.304975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.304956    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 16:09:43.305324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.305302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 16:09:43.305438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.305342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 16:09:43.305438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.305371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 16:09:43.305438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.305402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d4kl9\""
Jul 28 16:09:43.309335 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.309302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 16:09:43.314226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.314197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c6cf54b8-tbn6f"]
Jul 28 16:09:43.471800 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-trusted-ca-bundle\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-oauth-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxbr\" (UniqueName: \"kubernetes.io/projected/3aad0404-b197-4263-8438-f419bed36de6-kube-api-access-8bxbr\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-console-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-service-ca\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.471998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.472064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.472021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-oauth-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573214 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573214 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-oauth-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-trusted-ca-bundle\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-oauth-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxbr\" (UniqueName: \"kubernetes.io/projected/3aad0404-b197-4263-8438-f419bed36de6-kube-api-access-8bxbr\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-console-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.573419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.573398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-service-ca\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.574112 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.574081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-oauth-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.574226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.574169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-console-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.574226 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.574175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-service-ca\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.574320 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.574303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aad0404-b197-4263-8438-f419bed36de6-trusted-ca-bundle\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.575763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.575739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-oauth-config\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.575872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.575800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3aad0404-b197-4263-8438-f419bed36de6-console-serving-cert\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.583431 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.583411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxbr\" (UniqueName: \"kubernetes.io/projected/3aad0404-b197-4263-8438-f419bed36de6-kube-api-access-8bxbr\") pod \"console-77c6cf54b8-tbn6f\" (UID: \"3aad0404-b197-4263-8438-f419bed36de6\") " pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.614007 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.613973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:43.746960 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.746933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c6cf54b8-tbn6f"]
Jul 28 16:09:43.749627 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:09:43.749602    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aad0404_b197_4263_8438_f419bed36de6.slice/crio-0a9b9b9c2d62af96205eefaedb65317950b16510db31e24743b7e6c79f8418b0 WatchSource:0}: Error finding container 0a9b9b9c2d62af96205eefaedb65317950b16510db31e24743b7e6c79f8418b0: Status 404 returned error can't find the container with id 0a9b9b9c2d62af96205eefaedb65317950b16510db31e24743b7e6c79f8418b0
Jul 28 16:09:43.860199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.860116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6cf54b8-tbn6f" event={"ID":"3aad0404-b197-4263-8438-f419bed36de6","Type":"ContainerStarted","Data":"ab579410be44a5acf5d99aefd4860835e4b4708267325d9cb06d2a481e92b320"}
Jul 28 16:09:43.860199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.860165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6cf54b8-tbn6f" event={"ID":"3aad0404-b197-4263-8438-f419bed36de6","Type":"ContainerStarted","Data":"0a9b9b9c2d62af96205eefaedb65317950b16510db31e24743b7e6c79f8418b0"}
Jul 28 16:09:43.879517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:43.879468    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77c6cf54b8-tbn6f" podStartSLOduration=0.879451306 podStartE2EDuration="879.451306ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:43.877718338 +0000 UTC m=+430.429316332" watchObservedRunningTime="2026-07-28 16:09:43.879451306 +0000 UTC m=+430.431049299"
Jul 28 16:09:53.614952 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:53.614918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:53.615337 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:53.614969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:53.619596 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:53.619570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:09:53.903999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:09:53.903918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77c6cf54b8-tbn6f"
Jul 28 16:10:09.628959 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.628928    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-tx9nx"]
Jul 28 16:10:09.632176 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.632162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.634424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.634401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 16:10:09.634938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.634920    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 16:10:09.635045 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.634935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4b7lq\""
Jul 28 16:10:09.635045 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.634935    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 16:10:09.641444 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.641427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tx9nx"]
Jul 28 16:10:09.676520 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.676493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.676676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.676535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbkv\" (UniqueName: \"kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.777418 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.777386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.777611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.777430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbkv\" (UniqueName: \"kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.777810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.777787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.786446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.786418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbkv\" (UniqueName: \"kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv\") pod \"s3-init-tx9nx\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") " pod="kserve/s3-init-tx9nx"
Jul 28 16:10:09.954324 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:09.954294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tx9nx"
Jul 28 16:10:10.087071 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:10.087038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tx9nx"]
Jul 28 16:10:10.089947 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:10:10.089919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf624a898_05de_4aa7_993a_8790c6c035b3.slice/crio-1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1 WatchSource:0}: Error finding container 1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1: Status 404 returned error can't find the container with id 1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1
Jul 28 16:10:10.091921 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:10.091905    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:10:10.964872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:10.964812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerStarted","Data":"1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1"}
Jul 28 16:10:15.989861 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:15.989805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerStarted","Data":"bfb2450c477ef9ec6d22a78353fd0898d165a81e0a4c5ab8305f3cba38ac208d"}
Jul 28 16:10:17.998480 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:17.998448    2579 generic.go:356] "Generic (PLEG): container finished" podID="f624a898-05de-4aa7-993a-8790c6c035b3" containerID="bfb2450c477ef9ec6d22a78353fd0898d165a81e0a4c5ab8305f3cba38ac208d" exitCode=0
Jul 28 16:10:17.998835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:17.998521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerDied","Data":"bfb2450c477ef9ec6d22a78353fd0898d165a81e0a4c5ab8305f3cba38ac208d"}
Jul 28 16:10:23.023029 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:23.022987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerStarted","Data":"c4b0e7d170167cecc5845fe10df421ed0a1169de7e682313e1ff60e6aaf38870"}
Jul 28 16:10:23.040806 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:10:23.040757    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-tx9nx" podStartSLOduration=1.832507531 podStartE2EDuration="14.040739833s" totalImagesPullingTime="9.73432649s" totalInitContainerRuntime="2.473905812s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:10:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:10:23.039549556 +0000 UTC m=+469.591147583" watchObservedRunningTime="2026-07-28 16:10:23.040739833 +0000 UTC m=+469.592337826"
Jul 28 16:11:28.282032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:28.281998    2579 generic.go:356] "Generic (PLEG): container finished" podID="f624a898-05de-4aa7-993a-8790c6c035b3" containerID="c4b0e7d170167cecc5845fe10df421ed0a1169de7e682313e1ff60e6aaf38870" exitCode=0
Jul 28 16:11:28.282433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:28.282072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerDied","Data":"c4b0e7d170167cecc5845fe10df421ed0a1169de7e682313e1ff60e6aaf38870"}
Jul 28 16:11:29.408498 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.408474    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tx9nx"
Jul 28 16:11:29.589830 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.589747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqbkv\" (UniqueName: \"kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv\") pod \"f624a898-05de-4aa7-993a-8790c6c035b3\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") "
Jul 28 16:11:29.589830 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.589802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache\") pod \"f624a898-05de-4aa7-993a-8790c6c035b3\" (UID: \"f624a898-05de-4aa7-993a-8790c6c035b3\") "
Jul 28 16:11:29.591965 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.591934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv" (OuterVolumeSpecName: "kube-api-access-nqbkv") pod "f624a898-05de-4aa7-993a-8790c6c035b3" (UID: "f624a898-05de-4aa7-993a-8790c6c035b3"). InnerVolumeSpecName "kube-api-access-nqbkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:11:29.608462 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.608436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f624a898-05de-4aa7-993a-8790c6c035b3" (UID: "f624a898-05de-4aa7-993a-8790c6c035b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:11:29.691291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.691262    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqbkv\" (UniqueName: \"kubernetes.io/projected/f624a898-05de-4aa7-993a-8790c6c035b3-kube-api-access-nqbkv\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:11:29.691291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:29.691290    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f624a898-05de-4aa7-993a-8790c6c035b3-model-cache\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:11:30.291001 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:30.290922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tx9nx" event={"ID":"f624a898-05de-4aa7-993a-8790c6c035b3","Type":"ContainerDied","Data":"1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1"}
Jul 28 16:11:30.291001 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:30.290965    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cfab033424ce1f5bc3891b477e76c173e59092ca39e003e4139e4af83a478a1"
Jul 28 16:11:30.291001 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:30.290931    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tx9nx"
Jul 28 16:11:42.982861 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.982812    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:11:42.983272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.983135    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f624a898-05de-4aa7-993a-8790c6c035b3" containerName="download-hf-model"
Jul 28 16:11:42.983272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.983146    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f624a898-05de-4aa7-993a-8790c6c035b3" containerName="download-hf-model"
Jul 28 16:11:42.983272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.983155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f624a898-05de-4aa7-993a-8790c6c035b3" containerName="s3-init"
Jul 28 16:11:42.983272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.983161    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f624a898-05de-4aa7-993a-8790c6c035b3" containerName="s3-init"
Jul 28 16:11:42.983272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.983219    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f624a898-05de-4aa7-993a-8790c6c035b3" containerName="s3-init"
Jul 28 16:11:42.986625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.986605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:42.989976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.989952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-3a741-predictor-serving-cert\""
Jul 28 16:11:42.990121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.989990    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:11:42.990121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.990004    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:11:42.990121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.990063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-3a741-kube-rbac-proxy-sar-config\""
Jul 28 16:11:42.990121 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.989995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:11:42.990410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.990394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:11:42.990465 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.990449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:11:42.997565 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:42.997545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:11:43.100130 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.100089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.100341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.100199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.100341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.100263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnfq\" (UniqueName: \"kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.100341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.100291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.100341 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.100307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.201567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.201527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.201786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.201583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnfq\" (UniqueName: \"kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.201786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.201622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.201786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.201645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.201786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.201711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.202363 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:11:43.201788    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-3a741-predictor-serving-cert" not found
Jul 28 16:11:43.202363 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:11:43.201923    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls podName:32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca nodeName:}" failed. No retries permitted until 2026-07-28 16:11:43.701885729 +0000 UTC m=+550.253483709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls") pod "isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca") : secret "isvc-raw-sklearn-batcher-3a741-predictor-serving-cert" not found
Jul 28 16:11:43.202363 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.202098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.202520 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.202394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.202520 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.202394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.211784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.211756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnfq\" (UniqueName: \"kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.706695 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.706654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.709080 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.709048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:43.896969 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:43.896920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:11:44.030585 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:44.030566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:11:44.032969 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:11:44.032944    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e4e0c9_dd2e_4dd9_90f2_6a204a7e27ca.slice/crio-c729ec6989a187e77bede495104164378cb091db170d7e3db21b9037aee73649 WatchSource:0}: Error finding container c729ec6989a187e77bede495104164378cb091db170d7e3db21b9037aee73649: Status 404 returned error can't find the container with id c729ec6989a187e77bede495104164378cb091db170d7e3db21b9037aee73649
Jul 28 16:11:44.345965 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:44.345882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerStarted","Data":"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"}
Jul 28 16:11:44.345965 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:44.345959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerStarted","Data":"c729ec6989a187e77bede495104164378cb091db170d7e3db21b9037aee73649"}
Jul 28 16:11:48.363461 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:48.363423    2579 generic.go:356] "Generic (PLEG): container finished" podID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerID="7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b" exitCode=0
Jul 28 16:11:48.363841 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:11:48.363497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerDied","Data":"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"}
Jul 28 16:12:01.436720 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:01.436683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerStarted","Data":"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"}
Jul 28 16:12:04.454980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:04.454938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerStarted","Data":"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"}
Jul 28 16:12:06.465559 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:06.465520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerStarted","Data":"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"}
Jul 28 16:12:06.465980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:06.465611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:12:06.498557 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:06.498466    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podStartSLOduration=3.146469485 podStartE2EDuration="24.498450661s" totalImagesPullingTime="17.741693993s" totalInitContainerRuntime="3.610287183s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:11:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:12:06.49549215 +0000 UTC m=+573.047090168" watchObservedRunningTime="2026-07-28 16:12:06.498450661 +0000 UTC m=+573.050048654"
Jul 28 16:12:07.470118 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:07.470080    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:12:07.470490 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:07.470146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:12:07.471572 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:07.471538    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:07.472263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:07.472237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:08.474516 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:08.474471    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:08.474979 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:08.474932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:08.478400 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:08.478378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:12:09.478931 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:09.478888    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:09.479373 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:09.479184    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:19.479638 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:19.479586    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:19.480201 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:19.480059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:29.479291 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:29.479238    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:29.479742 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:29.479689    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:33.856768 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:33.856740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:12:33.857221 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:33.856936    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:12:39.479349 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:39.479294    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:39.479767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:39.479739    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:49.479171 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:49.479121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:49.479617 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:49.479452    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:59.478973 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:59.478841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:12:59.479432 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:12:59.479285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:09.479183 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:09.479132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:13:09.479625 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:09.479601    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:19.480055 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:19.480018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:19.481335 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:19.480153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:28.379462 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.379429    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:13:28.379926 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.379678    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" containerID="cri-o://9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023" gracePeriod=30
Jul 28 16:13:28.379926 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.379691    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" containerID="cri-o://60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c" gracePeriod=30
Jul 28 16:13:28.379926 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.379746    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" containerID="cri-o://cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7" gracePeriod=30
Jul 28 16:13:28.408269 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.408237    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:13:28.410664 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.410645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.415525 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.415499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8422a-predictor-serving-cert\""
Jul 28 16:13:28.416955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.416932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8422a-kube-rbac-proxy-sar-config\""
Jul 28 16:13:28.431455 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.431431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:13:28.474961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.474927    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:28.553907 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.553878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.554105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.553921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.554105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.553962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqzx\" (UniqueName: \"kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.554105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.554056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.554105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.554093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.611987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.611954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:13:28.614365 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.614337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.616733 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.616706    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8422a-kube-rbac-proxy-sar-config\""
Jul 28 16:13:28.616892 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.616706    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8422a-predictor-serving-cert\""
Jul 28 16:13:28.635541 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.635486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:13:28.654907 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.654878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.654907 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.654908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.654952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.654988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.655025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqzx\" (UniqueName: \"kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655360 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.655336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655644 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.655620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.655704 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.655668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.657579 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.657559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.667157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.667135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqzx\" (UniqueName: \"kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx\") pod \"isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.720790 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.720760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:28.756335 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.756305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.756488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.756341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqlf\" (UniqueName: \"kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.756488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.756387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.756488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.756413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.756488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.756431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.803746 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.803706    2579 generic.go:356] "Generic (PLEG): container finished" podID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerID="cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7" exitCode=2
Jul 28 16:13:28.803899 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.803757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerDied","Data":"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"}
Jul 28 16:13:28.857413 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.857377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.857568 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.857420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqlf\" (UniqueName: \"kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.857568 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.857471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.857660 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.857630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.857709 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.857673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.858085 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.858060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.858197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.858131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.858237 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.858197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.859829 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.859813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.868699 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.868650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:13:28.871303 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:13:28.871284    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e06ade3_d876_4927_8bc4_92aabdf64b17.slice/crio-19b25a80b5e1f4f108738bc5b5a8fe3aa966fc7fb78db30e45db99603ca96894 WatchSource:0}: Error finding container 19b25a80b5e1f4f108738bc5b5a8fe3aa966fc7fb78db30e45db99603ca96894: Status 404 returned error can't find the container with id 19b25a80b5e1f4f108738bc5b5a8fe3aa966fc7fb78db30e45db99603ca96894
Jul 28 16:13:28.874943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.874915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqlf\" (UniqueName: \"kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf\") pod \"isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:28.924387 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:28.924363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:29.074107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.074076    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:13:29.077002 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:13:29.076975    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde5f140_8d29_41d4_9aa6_9cf62d5fc16f.slice/crio-ce3d75eae990f6d8f9e2349213836db7e9c807c867c71f5a5d75d4490951b8fb WatchSource:0}: Error finding container ce3d75eae990f6d8f9e2349213836db7e9c807c867c71f5a5d75d4490951b8fb: Status 404 returned error can't find the container with id ce3d75eae990f6d8f9e2349213836db7e9c807c867c71f5a5d75d4490951b8fb
Jul 28 16:13:29.479407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.479368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:13:29.479906 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.479710    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:29.811253 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.811152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerStarted","Data":"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"}
Jul 28 16:13:29.811405 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.811256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerStarted","Data":"19b25a80b5e1f4f108738bc5b5a8fe3aa966fc7fb78db30e45db99603ca96894"}
Jul 28 16:13:29.812721 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.812689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerStarted","Data":"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"}
Jul 28 16:13:29.812868 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:29.812751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerStarted","Data":"ce3d75eae990f6d8f9e2349213836db7e9c807c867c71f5a5d75d4490951b8fb"}
Jul 28 16:13:32.830059 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:32.829985    2579 generic.go:356] "Generic (PLEG): container finished" podID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerID="c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d" exitCode=0
Jul 28 16:13:32.830422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:32.830064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerDied","Data":"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"}
Jul 28 16:13:32.831902 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:32.831782    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerID="c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333" exitCode=0
Jul 28 16:13:32.831902 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:32.831839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerDied","Data":"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"}
Jul 28 16:13:33.475317 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.475267    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:33.841885 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.841518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerStarted","Data":"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"}
Jul 28 16:13:33.841885 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.841646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerStarted","Data":"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"}
Jul 28 16:13:33.842386 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.842241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:33.842445 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.842391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:33.844119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.843966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:13:33.844925 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.844893    2579 generic.go:356] "Generic (PLEG): container finished" podID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerID="9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023" exitCode=0
Jul 28 16:13:33.845028 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.845010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerDied","Data":"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"}
Jul 28 16:13:33.865437 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:33.865377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podStartSLOduration=2.221279052 podStartE2EDuration="5.865357233s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644078181s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:13:33.863680437 +0000 UTC m=+660.415278430" watchObservedRunningTime="2026-07-28 16:13:33.865357233 +0000 UTC m=+660.416955226"
Jul 28 16:13:34.851989 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:34.851942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:13:35.857079 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:35.857033    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:13:38.474810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:38.474771    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:38.475176 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:38.474903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:39.479697 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:39.479648    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:13:39.480197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:39.480079    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:40.862660 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:40.862631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:13:40.863240 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:40.863214    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:13:43.475138 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:43.475097    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:48.475176 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:48.475131    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:49.479194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:49.479145    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:13:49.479689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:49.479302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:49.479689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:49.479514    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:49.479689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:49.479616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:50.864142 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:50.864102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:13:52.943528 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:52.943495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerStarted","Data":"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"}
Jul 28 16:13:52.943936 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:52.943557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerStarted","Data":"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"}
Jul 28 16:13:52.943936 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:52.943756    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:52.973810 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:52.973765    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podStartSLOduration=1.749491703 podStartE2EDuration="24.973750346s" totalImagesPullingTime="19.56606374s" totalInitContainerRuntime="3.658194903s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:13:52.972870428 +0000 UTC m=+679.524468421" watchObservedRunningTime="2026-07-28 16:13:52.973750346 +0000 UTC m=+679.525348339"
Jul 28 16:13:53.474944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:53.474906    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:13:53.947948 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:53.947915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:53.949300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:53.949275    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:13:54.952581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:54.952542    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:13:58.533732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.533705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:58.651550 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config\") pod \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") "
Jul 28 16:13:58.651550 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert\") pod \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") "
Jul 28 16:13:58.651550 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") pod \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") "
Jul 28 16:13:58.651826 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxnfq\" (UniqueName: \"kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq\") pod \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") "
Jul 28 16:13:58.651826 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location\") pod \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\" (UID: \"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca\") "
Jul 28 16:13:58.651826 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:13:58.652006 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651904    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:13:58.652006 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.651976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:13:58.652080 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.652002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:13:58.653829 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.653809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:13:58.653940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.653878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq" (OuterVolumeSpecName: "kube-api-access-bxnfq") pod "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" (UID: "32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca"). InnerVolumeSpecName "kube-api-access-bxnfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:13:58.753103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.753068    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:13:58.753103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.753097    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:13:58.753103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.753107    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:13:58.753321 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.753120    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnfq\" (UniqueName: \"kubernetes.io/projected/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca-kube-api-access-bxnfq\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:13:58.972438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.972412    2579 generic.go:356] "Generic (PLEG): container finished" podID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerID="60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c" exitCode=0
Jul 28 16:13:58.972607 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.972463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerDied","Data":"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"}
Jul 28 16:13:58.972607 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.972473    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"
Jul 28 16:13:58.972607 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.972518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" event={"ID":"32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca","Type":"ContainerDied","Data":"c729ec6989a187e77bede495104164378cb091db170d7e3db21b9037aee73649"}
Jul 28 16:13:58.972607 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.972550    2579 scope.go:117] "RemoveContainer" containerID="60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"
Jul 28 16:13:58.981377 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.981356    2579 scope.go:117] "RemoveContainer" containerID="cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"
Jul 28 16:13:58.994776 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.994757    2579 scope.go:117] "RemoveContainer" containerID="9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"
Jul 28 16:13:58.999788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:58.999765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:13:59.002282 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.002266    2579 scope.go:117] "RemoveContainer" containerID="7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"
Jul 28 16:13:59.004371 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.004351    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7"]
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.009839    2579 scope.go:117] "RemoveContainer" containerID="60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:13:59.010258    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c\": container with ID starting with 60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c not found: ID does not exist" containerID="60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010289    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c"} err="failed to get container status \"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c\": rpc error: code = NotFound desc = could not find container \"60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c\": container with ID starting with 60327bae739b55a615812d0f7adf7bd2a9df8f6ace7c3d86ed16c3e3471c675c not found: ID does not exist"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010312    2579 scope.go:117] "RemoveContainer" containerID="cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:13:59.010570    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7\": container with ID starting with cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7 not found: ID does not exist" containerID="cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010591    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7"} err="failed to get container status \"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7\": rpc error: code = NotFound desc = could not find container \"cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7\": container with ID starting with cca3bf8a113cecb32685d49f03c5e3564c5671fb1dde2c6dd4809fc5bea34cb7 not found: ID does not exist"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010608    2579 scope.go:117] "RemoveContainer" containerID="9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:13:59.010901    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023\": container with ID starting with 9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023 not found: ID does not exist" containerID="9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010922    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023"} err="failed to get container status \"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023\": rpc error: code = NotFound desc = could not find container \"9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023\": container with ID starting with 9e6ecdbc198bc9dcb6393e7fd1f8841e61119d3e11bf340d6427fba497837023 not found: ID does not exist"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.010940    2579 scope.go:117] "RemoveContainer" containerID="7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:13:59.011214    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b\": container with ID starting with 7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b not found: ID does not exist" containerID="7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"
Jul 28 16:13:59.012285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.011235    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b"} err="failed to get container status \"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b\": rpc error: code = NotFound desc = could not find container \"7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b\": container with ID starting with 7f694c7f9ace202570631702671258d3bdb20c80a419a541ea375443d2e5b05b not found: ID does not exist"
Jul 28 16:13:59.475485 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.475445    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3a741-predictor-7599c9c8c6-2j7k7" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 16:13:59.957397 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.957368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:13:59.958075 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:13:59.958045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:00.014777 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:00.014741    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" path="/var/lib/kubelet/pods/32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca/volumes"
Jul 28 16:14:00.863110 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:00.863071    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:09.958011 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:09.957973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:10.863356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:10.863313    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:19.957992 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:19.957946    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:20.863614 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:20.863576    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:29.958157 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:29.958113    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:30.863367 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:30.863331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:39.958459 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:39.958416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:40.863303 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:40.863262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:43.009573 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:43.009530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:14:49.958901 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:49.958826    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:14:53.010647 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:53.010603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:14:59.958566 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:14:59.958534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:15:18.754748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.754713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755056    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755067    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755088    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755094    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755102    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="storage-initializer"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755108    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="storage-initializer"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755115    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755122    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755190    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="agent"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755202    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kserve-container"
Jul 28 16:15:18.755242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.755208    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e4e0c9-dd2e-4dd9-90f2-6a204a7e27ca" containerName="kube-rbac-proxy"
Jul 28 16:15:18.757351 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.757331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.760011 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.759989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-69837-predictor-serving-cert\""
Jul 28 16:15:18.760129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.760023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-69837-kube-rbac-proxy-sar-config\""
Jul 28 16:15:18.774123 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.774088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:15:18.784516 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.784488    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:15:18.784753 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.784728    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" containerID="cri-o://c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f" gracePeriod=30
Jul 28 16:15:18.784827 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.784734    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kube-rbac-proxy" containerID="cri-o://c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697" gracePeriod=30
Jul 28 16:15:18.851954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.851913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.852116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.851966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9lq\" (UniqueName: \"kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.852116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.851996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.852116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.852027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.852116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.852106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.932129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.932099    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:15:18.935194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.935177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.937668 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.937645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-69837-predictor-serving-cert\""
Jul 28 16:15:18.937757 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.937705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-69837-kube-rbac-proxy-sar-config\""
Jul 28 16:15:18.951079 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.951053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:15:18.953068 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.953186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9lq\" (UniqueName: \"kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.953186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.953186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.953186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.953410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.953410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.953410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cn9z\" (UniqueName: \"kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.953410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.953410 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:18.953679 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.953531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.954106 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.954079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.954213 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.954189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.955973 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.955942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.967239 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.967194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9lq\" (UniqueName: \"kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq\") pod \"isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:18.982513 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.982488    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:15:18.982788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.982743    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" containerID="cri-o://9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610" gracePeriod=30
Jul 28 16:15:18.982915 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:18.982759    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kube-rbac-proxy" containerID="cri-o://4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a" gracePeriod=30
Jul 28 16:15:19.054556 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054556 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054556 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cn9z\" (UniqueName: \"kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054820 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054820 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054955 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.054910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.054955 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:19.054936    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-69837-predictor-serving-cert" not found
Jul 28 16:15:19.055053 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:19.055013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls podName:099007c1-560e-4197-8284-ec00886361ac nodeName:}" failed. No retries permitted until 2026-07-28 16:15:19.554990463 +0000 UTC m=+766.106588438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" (UID: "099007c1-560e-4197-8284-ec00886361ac") : secret "isvc-xgboost-graph-raw-hpa-69837-predictor-serving-cert" not found
Jul 28 16:15:19.055301 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.055284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.055424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.055405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.064234 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.064212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cn9z\" (UniqueName: \"kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.070049 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.070023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:19.206877 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.206838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:15:19.209282 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:15:19.209260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aa27f7_e78c_4ffe_a756_61b34c82ebe5.slice/crio-d00c53e4928e801664287153a17ffc43bb419406637a88e7b1cae0eacde73f69 WatchSource:0}: Error finding container d00c53e4928e801664287153a17ffc43bb419406637a88e7b1cae0eacde73f69: Status 404 returned error can't find the container with id d00c53e4928e801664287153a17ffc43bb419406637a88e7b1cae0eacde73f69
Jul 28 16:15:19.329441 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.329356    2579 generic.go:356] "Generic (PLEG): container finished" podID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerID="4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a" exitCode=2
Jul 28 16:15:19.329441 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.329421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerDied","Data":"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"}
Jul 28 16:15:19.330970 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.330936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerStarted","Data":"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"}
Jul 28 16:15:19.331101 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.331003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerStarted","Data":"d00c53e4928e801664287153a17ffc43bb419406637a88e7b1cae0eacde73f69"}
Jul 28 16:15:19.333023 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.332995    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerID="c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697" exitCode=2
Jul 28 16:15:19.333112 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.333033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerDied","Data":"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"}
Jul 28 16:15:19.559264 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.559227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.561710 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.561675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.845772 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.845734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:19.953466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.953424    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 16:15:19.958494 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.958457    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:15:19.977947 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:19.977926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:15:19.980569 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:15:19.980538    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099007c1_560e_4197_8284_ec00886361ac.slice/crio-68fb5a0ad8b6ae7a2d6b9f58df22ef9f60b018d1272df9977d17b9477b16fbd2 WatchSource:0}: Error finding container 68fb5a0ad8b6ae7a2d6b9f58df22ef9f60b018d1272df9977d17b9477b16fbd2: Status 404 returned error can't find the container with id 68fb5a0ad8b6ae7a2d6b9f58df22ef9f60b018d1272df9977d17b9477b16fbd2
Jul 28 16:15:20.339723 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:20.339687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerStarted","Data":"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"}
Jul 28 16:15:20.339936 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:20.339761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerStarted","Data":"68fb5a0ad8b6ae7a2d6b9f58df22ef9f60b018d1272df9977d17b9477b16fbd2"}
Jul 28 16:15:20.857659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:20.857619    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 16:15:22.630376 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.630351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:15:22.687738 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.687711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcqlf\" (UniqueName: \"kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf\") pod \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") "
Jul 28 16:15:22.687953 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.687755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls\") pod \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") "
Jul 28 16:15:22.687953 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.687786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config\") pod \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") "
Jul 28 16:15:22.687953 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.687838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert\") pod \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") "
Jul 28 16:15:22.687953 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.687881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location\") pod \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\" (UID: \"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f\") "
Jul 28 16:15:22.688229 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.688188    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" (UID: "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:15:22.688229 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.688214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" (UID: "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:22.688385 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.688196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" (UID: "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:22.689975 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.689952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" (UID: "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:22.690048 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.690011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf" (OuterVolumeSpecName: "kube-api-access-bcqlf") pod "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" (UID: "cde5f140-8d29-41d4-9aa6-9cf62d5fc16f"). InnerVolumeSpecName "kube-api-access-bcqlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:15:22.788961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.788884    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:22.788961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.788912    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:22.788961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.788922    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqlf\" (UniqueName: \"kubernetes.io/projected/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-api-access-bcqlf\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:22.788961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.788934    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:22.788961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:22.788944    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:23.009993 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.009950    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:15:23.355295 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.355206    2579 generic.go:356] "Generic (PLEG): container finished" podID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerID="e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b" exitCode=0
Jul 28 16:15:23.355468 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.355287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerDied","Data":"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"}
Jul 28 16:15:23.357298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.357277    2579 generic.go:356] "Generic (PLEG): container finished" podID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerID="9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610" exitCode=0
Jul 28 16:15:23.357427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.357318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerDied","Data":"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"}
Jul 28 16:15:23.357427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.357332    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"
Jul 28 16:15:23.357427 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.357360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp" event={"ID":"cde5f140-8d29-41d4-9aa6-9cf62d5fc16f","Type":"ContainerDied","Data":"ce3d75eae990f6d8f9e2349213836db7e9c807c867c71f5a5d75d4490951b8fb"}
Jul 28 16:15:23.357552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.357440    2579 scope.go:117] "RemoveContainer" containerID="4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"
Jul 28 16:15:23.369516 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.369498    2579 scope.go:117] "RemoveContainer" containerID="9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"
Jul 28 16:15:23.377355 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.377318    2579 scope.go:117] "RemoveContainer" containerID="c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"
Jul 28 16:15:23.391453 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.391329    2579 scope.go:117] "RemoveContainer" containerID="4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"
Jul 28 16:15:23.391652 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:23.391627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a\": container with ID starting with 4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a not found: ID does not exist" containerID="4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"
Jul 28 16:15:23.391767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.391661    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a"} err="failed to get container status \"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a\": rpc error: code = NotFound desc = could not find container \"4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a\": container with ID starting with 4f519775038774be7600208e212ae1a624a8f7c1461ed63b5249e89f3c2ef60a not found: ID does not exist"
Jul 28 16:15:23.391767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.391685    2579 scope.go:117] "RemoveContainer" containerID="9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"
Jul 28 16:15:23.392004 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:23.391986    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610\": container with ID starting with 9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610 not found: ID does not exist" containerID="9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"
Jul 28 16:15:23.392075 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.392009    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610"} err="failed to get container status \"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610\": rpc error: code = NotFound desc = could not find container \"9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610\": container with ID starting with 9c28ed20e2958df752a91c36bf2cdeba8016a20b0f0df9a190b92648c7d85610 not found: ID does not exist"
Jul 28 16:15:23.392075 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.392025    2579 scope.go:117] "RemoveContainer" containerID="c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"
Jul 28 16:15:23.392325 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:23.392303    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d\": container with ID starting with c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d not found: ID does not exist" containerID="c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"
Jul 28 16:15:23.392393 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.392334    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d"} err="failed to get container status \"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d\": rpc error: code = NotFound desc = could not find container \"c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d\": container with ID starting with c796545288fda854ba5bac678eff813fd239362e5217f46f14944cc9ece0ef5d not found: ID does not exist"
Jul 28 16:15:23.393334 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.393313    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:15:23.405173 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.405152    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8422a-predictor-7548cfc697-rz6pp"]
Jul 28 16:15:23.848732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.848708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:15:23.900873 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.900766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location\") pod \"5e06ade3-d876-4927-8bc4-92aabdf64b17\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") "
Jul 28 16:15:23.901039 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.900867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert\") pod \"5e06ade3-d876-4927-8bc4-92aabdf64b17\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") "
Jul 28 16:15:23.901039 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.900912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsqzx\" (UniqueName: \"kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx\") pod \"5e06ade3-d876-4927-8bc4-92aabdf64b17\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") "
Jul 28 16:15:23.901177 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config\") pod \"5e06ade3-d876-4927-8bc4-92aabdf64b17\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") "
Jul 28 16:15:23.901177 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e06ade3-d876-4927-8bc4-92aabdf64b17" (UID: "5e06ade3-d876-4927-8bc4-92aabdf64b17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:15:23.901177 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls\") pod \"5e06ade3-d876-4927-8bc4-92aabdf64b17\" (UID: \"5e06ade3-d876-4927-8bc4-92aabdf64b17\") "
Jul 28 16:15:23.901375 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901328    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e06ade3-d876-4927-8bc4-92aabdf64b17" (UID: "5e06ade3-d876-4927-8bc4-92aabdf64b17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:23.901438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e06ade3-d876-4927-8bc4-92aabdf64b17" (UID: "5e06ade3-d876-4927-8bc4-92aabdf64b17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:23.901488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901445    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:23.901488 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.901463    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e06ade3-d876-4927-8bc4-92aabdf64b17-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:23.903060 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.903032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx" (OuterVolumeSpecName: "kube-api-access-qsqzx") pod "5e06ade3-d876-4927-8bc4-92aabdf64b17" (UID: "5e06ade3-d876-4927-8bc4-92aabdf64b17"). InnerVolumeSpecName "kube-api-access-qsqzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:15:23.903262 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:23.903239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e06ade3-d876-4927-8bc4-92aabdf64b17" (UID: "5e06ade3-d876-4927-8bc4-92aabdf64b17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:24.002184 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.002151    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e06ade3-d876-4927-8bc4-92aabdf64b17-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:24.002184 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.002179    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqzx\" (UniqueName: \"kubernetes.io/projected/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-api-access-qsqzx\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:24.002184 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.002188    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e06ade3-d876-4927-8bc4-92aabdf64b17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:15:24.020077 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.020047    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" path="/var/lib/kubelet/pods/cde5f140-8d29-41d4-9aa6-9cf62d5fc16f/volumes"
Jul 28 16:15:24.367523 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.367493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerStarted","Data":"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"}
Jul 28 16:15:24.367754 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.367556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerStarted","Data":"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"}
Jul 28 16:15:24.367905 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.367882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:24.368052 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.368031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:24.369076 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.369053    2579 generic.go:356] "Generic (PLEG): container finished" podID="099007c1-560e-4197-8284-ec00886361ac" containerID="c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172" exitCode=0
Jul 28 16:15:24.369183 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.369093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerDied","Data":"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"}
Jul 28 16:15:24.369611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.369588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:15:24.370963 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.370941    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerID="c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f" exitCode=0
Jul 28 16:15:24.371133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.370984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerDied","Data":"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"}
Jul 28 16:15:24.371133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.370994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"
Jul 28 16:15:24.371133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.371039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt" event={"ID":"5e06ade3-d876-4927-8bc4-92aabdf64b17","Type":"ContainerDied","Data":"19b25a80b5e1f4f108738bc5b5a8fe3aa966fc7fb78db30e45db99603ca96894"}
Jul 28 16:15:24.371133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.371080    2579 scope.go:117] "RemoveContainer" containerID="c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"
Jul 28 16:15:24.381244 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.381226    2579 scope.go:117] "RemoveContainer" containerID="c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"
Jul 28 16:15:24.388158 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.388031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podStartSLOduration=2.605337716 podStartE2EDuration="6.388015025s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782677309s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:15:24.387143136 +0000 UTC m=+770.938741140" watchObservedRunningTime="2026-07-28 16:15:24.388015025 +0000 UTC m=+770.939613021"
Jul 28 16:15:24.390376 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.390355    2579 scope.go:117] "RemoveContainer" containerID="c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"
Jul 28 16:15:24.402467 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.402446    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:15:24.406567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.406550    2579 scope.go:117] "RemoveContainer" containerID="c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"
Jul 28 16:15:24.406877 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:24.406821    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697\": container with ID starting with c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697 not found: ID does not exist" containerID="c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"
Jul 28 16:15:24.406977 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.406878    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697"} err="failed to get container status \"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697\": rpc error: code = NotFound desc = could not find container \"c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697\": container with ID starting with c235f3b3ecacd6fb936c072230f88cab83c11ac01ad04b99f865c8661129b697 not found: ID does not exist"
Jul 28 16:15:24.406977 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.406902    2579 scope.go:117] "RemoveContainer" containerID="c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"
Jul 28 16:15:24.407381 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:24.407197    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f\": container with ID starting with c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f not found: ID does not exist" containerID="c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"
Jul 28 16:15:24.407381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.407228    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f"} err="failed to get container status \"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f\": rpc error: code = NotFound desc = could not find container \"c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f\": container with ID starting with c66204149b8308775bd7f9944c158aef4c11d59052df22c0e412000bd29f116f not found: ID does not exist"
Jul 28 16:15:24.407381 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.407249    2579 scope.go:117] "RemoveContainer" containerID="c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"
Jul 28 16:15:24.407583 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:15:24.407515    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333\": container with ID starting with c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333 not found: ID does not exist" containerID="c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"
Jul 28 16:15:24.407583 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.407541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333"} err="failed to get container status \"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333\": rpc error: code = NotFound desc = could not find container \"c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333\": container with ID starting with c40e04fb5d3574e1f5d368d8415694574c918e26604ba579e3ffde3d4594a333 not found: ID does not exist"
Jul 28 16:15:24.409419 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:24.409400    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8422a-predictor-559c6d8d96-b9mjt"]
Jul 28 16:15:25.377420 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:25.377386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerStarted","Data":"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"}
Jul 28 16:15:25.377866 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:25.377458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerStarted","Data":"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"}
Jul 28 16:15:25.377866 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:25.377684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:25.378689 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:25.378667    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:15:25.401734 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:25.401689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podStartSLOduration=3.655410636 podStartE2EDuration="7.401675403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746264767s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:15:25.3995196 +0000 UTC m=+771.951117592" watchObservedRunningTime="2026-07-28 16:15:25.401675403 +0000 UTC m=+771.953273396"
Jul 28 16:15:26.016275 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:26.016242    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" path="/var/lib/kubelet/pods/5e06ade3-d876-4927-8bc4-92aabdf64b17/volumes"
Jul 28 16:15:26.382611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:26.382528    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:26.383819 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:26.383790    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:15:27.388263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:27.388228    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:15:30.383594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:30.383568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:15:30.384078 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:30.384054    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:15:32.393445 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:32.393417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:15:32.393985 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:32.393963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:15:40.384519 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:40.384480    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:15:42.393938 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:42.393902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:15:50.384109 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:50.384065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:15:52.393839 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:15:52.393798    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:16:00.384238 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:00.384150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:16:02.394100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:02.394060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:16:10.384292 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:10.384243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:16:12.394803 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:12.394761    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:16:20.385044 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:20.385005    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:16:22.393908 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:22.393867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:16:30.384032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:30.383990    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:16:32.393997 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:32.393968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:16:40.384887 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:16:40.384828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:17:08.980191 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:08.980158    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:17:08.980767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:08.980486    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" containerID="cri-o://1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2" gracePeriod=30
Jul 28 16:17:08.980767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:08.980521    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kube-rbac-proxy" containerID="cri-o://9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423" gracePeriod=30
Jul 28 16:17:09.121667 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.121634    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:17:09.121964 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.121925    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" containerID="cri-o://7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f" gracePeriod=30
Jul 28 16:17:09.122054 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.121996    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kube-rbac-proxy" containerID="cri-o://3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd" gracePeriod=30
Jul 28 16:17:09.853266 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.853233    2579 generic.go:356] "Generic (PLEG): container finished" podID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerID="9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423" exitCode=2
Jul 28 16:17:09.853424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.853304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerDied","Data":"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"}
Jul 28 16:17:09.855290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.855270    2579 generic.go:356] "Generic (PLEG): container finished" podID="099007c1-560e-4197-8284-ec00886361ac" containerID="3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd" exitCode=2
Jul 28 16:17:09.855398 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:09.855341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerDied","Data":"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"}
Jul 28 16:17:10.379682 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:10.379639    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 16:17:10.384983 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:10.384956    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:17:12.388581 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.388534    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 16:17:12.393991 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.393957    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:17:12.867061 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.867038    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:17:12.869394 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.869369    2579 generic.go:356] "Generic (PLEG): container finished" podID="099007c1-560e-4197-8284-ec00886361ac" containerID="7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f" exitCode=0
Jul 28 16:17:12.869513 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.869422    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"
Jul 28 16:17:12.869513 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.869450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerDied","Data":"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"}
Jul 28 16:17:12.869586 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.869515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q" event={"ID":"099007c1-560e-4197-8284-ec00886361ac","Type":"ContainerDied","Data":"68fb5a0ad8b6ae7a2d6b9f58df22ef9f60b018d1272df9977d17b9477b16fbd2"}
Jul 28 16:17:12.869586 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.869556    2579 scope.go:117] "RemoveContainer" containerID="3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"
Jul 28 16:17:12.877251 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.877229    2579 scope.go:117] "RemoveContainer" containerID="7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"
Jul 28 16:17:12.884568 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.884551    2579 scope.go:117] "RemoveContainer" containerID="c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"
Jul 28 16:17:12.894829 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.894807    2579 scope.go:117] "RemoveContainer" containerID="3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"
Jul 28 16:17:12.895146 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:12.895096    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd\": container with ID starting with 3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd not found: ID does not exist" containerID="3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"
Jul 28 16:17:12.895146 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.895124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd"} err="failed to get container status \"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd\": rpc error: code = NotFound desc = could not find container \"3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd\": container with ID starting with 3c481a20784f706319ebe7307ca64f93032b2a897b7fae007ce8b3cb3ff240cd not found: ID does not exist"
Jul 28 16:17:12.895146 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.895142    2579 scope.go:117] "RemoveContainer" containerID="7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"
Jul 28 16:17:12.895382 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:12.895366    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f\": container with ID starting with 7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f not found: ID does not exist" containerID="7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"
Jul 28 16:17:12.895446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.895393    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f"} err="failed to get container status \"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f\": rpc error: code = NotFound desc = could not find container \"7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f\": container with ID starting with 7f1f45418ea4fbb8f2e2f86d235b7c34b48086448b430d4db1f06c06df00cd0f not found: ID does not exist"
Jul 28 16:17:12.895446 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.895406    2579 scope.go:117] "RemoveContainer" containerID="c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"
Jul 28 16:17:12.895677 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:12.895655    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172\": container with ID starting with c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172 not found: ID does not exist" containerID="c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"
Jul 28 16:17:12.895753 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.895681    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172"} err="failed to get container status \"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172\": rpc error: code = NotFound desc = could not find container \"c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172\": container with ID starting with c20284203cd67a2f8d55459891f8319c9db364bf5e7f46e656112e880faf3172 not found: ID does not exist"
Jul 28 16:17:12.936086 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936059    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert\") pod \"099007c1-560e-4197-8284-ec00886361ac\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") "
Jul 28 16:17:12.936211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936095    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cn9z\" (UniqueName: \"kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z\") pod \"099007c1-560e-4197-8284-ec00886361ac\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") "
Jul 28 16:17:12.936211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location\") pod \"099007c1-560e-4197-8284-ec00886361ac\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") "
Jul 28 16:17:12.936211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936163    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config\") pod \"099007c1-560e-4197-8284-ec00886361ac\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") "
Jul 28 16:17:12.936211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") pod \"099007c1-560e-4197-8284-ec00886361ac\" (UID: \"099007c1-560e-4197-8284-ec00886361ac\") "
Jul 28 16:17:12.936494 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "099007c1-560e-4197-8284-ec00886361ac" (UID: "099007c1-560e-4197-8284-ec00886361ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:17:12.936494 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "099007c1-560e-4197-8284-ec00886361ac" (UID: "099007c1-560e-4197-8284-ec00886361ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:17:12.936594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.936528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "099007c1-560e-4197-8284-ec00886361ac" (UID: "099007c1-560e-4197-8284-ec00886361ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:17:12.938259 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.938233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "099007c1-560e-4197-8284-ec00886361ac" (UID: "099007c1-560e-4197-8284-ec00886361ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:17:12.938349 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:12.938301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z" (OuterVolumeSpecName: "kube-api-access-8cn9z") pod "099007c1-560e-4197-8284-ec00886361ac" (UID: "099007c1-560e-4197-8284-ec00886361ac"). InnerVolumeSpecName "kube-api-access-8cn9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:17:13.037329 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.037292    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.037552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.037449    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cn9z\" (UniqueName: \"kubernetes.io/projected/099007c1-560e-4197-8284-ec00886361ac-kube-api-access-8cn9z\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.037552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.037469    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099007c1-560e-4197-8284-ec00886361ac-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.037552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.037478    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099007c1-560e-4197-8284-ec00886361ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.037552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.037488    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099007c1-560e-4197-8284-ec00886361ac-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.201006 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.200968    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:17:13.205804 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.205780    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-69837-predictor-594697479c-qsh7q"]
Jul 28 16:17:13.616003 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.615972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:17:13.642325 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config\") pod \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") "
Jul 28 16:17:13.642489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert\") pod \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") "
Jul 28 16:17:13.642489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location\") pod \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") "
Jul 28 16:17:13.642489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls\") pod \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") "
Jul 28 16:17:13.642489 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m9lq\" (UniqueName: \"kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq\") pod \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\" (UID: \"29aa27f7-e78c-4ffe-a756-61b34c82ebe5\") "
Jul 28 16:17:13.642835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29aa27f7-e78c-4ffe-a756-61b34c82ebe5" (UID: "29aa27f7-e78c-4ffe-a756-61b34c82ebe5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:17:13.642835 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29aa27f7-e78c-4ffe-a756-61b34c82ebe5" (UID: "29aa27f7-e78c-4ffe-a756-61b34c82ebe5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:17:13.642996 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.642858    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29aa27f7-e78c-4ffe-a756-61b34c82ebe5" (UID: "29aa27f7-e78c-4ffe-a756-61b34c82ebe5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:17:13.644605 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.644577    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq" (OuterVolumeSpecName: "kube-api-access-8m9lq") pod "29aa27f7-e78c-4ffe-a756-61b34c82ebe5" (UID: "29aa27f7-e78c-4ffe-a756-61b34c82ebe5"). InnerVolumeSpecName "kube-api-access-8m9lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:17:13.644706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.644624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29aa27f7-e78c-4ffe-a756-61b34c82ebe5" (UID: "29aa27f7-e78c-4ffe-a756-61b34c82ebe5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:17:13.743943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.743843    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.743943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.743890    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.743943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.743901    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9lq\" (UniqueName: \"kubernetes.io/projected/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-api-access-8m9lq\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.743943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.743912    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.743943 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.743920    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29aa27f7-e78c-4ffe-a756-61b34c82ebe5-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:17:13.876083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.876047    2579 generic.go:356] "Generic (PLEG): container finished" podID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerID="1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2" exitCode=0
Jul 28 16:17:13.876271 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.876124    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"
Jul 28 16:17:13.876271 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.876135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerDied","Data":"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"}
Jul 28 16:17:13.876271 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.876210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw" event={"ID":"29aa27f7-e78c-4ffe-a756-61b34c82ebe5","Type":"ContainerDied","Data":"d00c53e4928e801664287153a17ffc43bb419406637a88e7b1cae0eacde73f69"}
Jul 28 16:17:13.876271 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.876248    2579 scope.go:117] "RemoveContainer" containerID="9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"
Jul 28 16:17:13.886611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.886439    2579 scope.go:117] "RemoveContainer" containerID="1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"
Jul 28 16:17:13.895764 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.895741    2579 scope.go:117] "RemoveContainer" containerID="e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"
Jul 28 16:17:13.900958 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.900934    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:17:13.903222 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.903204    2579 scope.go:117] "RemoveContainer" containerID="9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"
Jul 28 16:17:13.903468 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:13.903446    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423\": container with ID starting with 9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423 not found: ID does not exist" containerID="9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"
Jul 28 16:17:13.903611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.903472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423"} err="failed to get container status \"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423\": rpc error: code = NotFound desc = could not find container \"9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423\": container with ID starting with 9bd03ede35a74aee8716f829fd7b9bd465a237423883e69d23213069d0834423 not found: ID does not exist"
Jul 28 16:17:13.903611 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.903488    2579 scope.go:117] "RemoveContainer" containerID="1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"
Jul 28 16:17:13.903799 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:13.903779    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2\": container with ID starting with 1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2 not found: ID does not exist" containerID="1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"
Jul 28 16:17:13.903928 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.903804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2"} err="failed to get container status \"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2\": rpc error: code = NotFound desc = could not find container \"1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2\": container with ID starting with 1e90ee97dba78ec6c6062da69e10ff2aed817b03a69045b2aaf1977eda4077c2 not found: ID does not exist"
Jul 28 16:17:13.903928 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.903823    2579 scope.go:117] "RemoveContainer" containerID="e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"
Jul 28 16:17:13.904321 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:13.904296    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b\": container with ID starting with e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b not found: ID does not exist" containerID="e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"
Jul 28 16:17:13.904405 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.904326    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b"} err="failed to get container status \"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b\": rpc error: code = NotFound desc = could not find container \"e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b\": container with ID starting with e30c89b990a6cc3aa8c64a263ee1f511dceff97a9a380bbd1125ff0b895cf17b not found: ID does not exist"
Jul 28 16:17:13.906288 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:13.906268    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-69837-predictor-8f66fd448-qnblw"]
Jul 28 16:17:14.014976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:14.014898    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099007c1-560e-4197-8284-ec00886361ac" path="/var/lib/kubelet/pods/099007c1-560e-4197-8284-ec00886361ac/volumes"
Jul 28 16:17:14.015398 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:14.015384    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" path="/var/lib/kubelet/pods/29aa27f7-e78c-4ffe-a756-61b34c82ebe5/volumes"
Jul 28 16:17:29.042283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042208    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042538    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042545    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042552    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042558    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042567    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042573    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042579    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042587    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="storage-initializer"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042594    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kube-rbac-proxy"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kube-rbac-proxy"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042609    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kube-rbac-proxy"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kube-rbac-proxy"
Jul 28 16:17:29.042630 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042634    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042642    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042678    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042685    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042704    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="storage-initializer"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042712    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="storage-initializer"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042724    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042732    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042798    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042811    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042822    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042829    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="099007c1-560e-4197-8284-ec00886361ac" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042835    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kserve-container"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042841    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde5f140-8d29-41d4-9aa6-9cf62d5fc16f" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042861    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e06ade3-d876-4927-8bc4-92aabdf64b17" containerName="kube-rbac-proxy"
Jul 28 16:17:29.043242 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.042870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29aa27f7-e78c-4ffe-a756-61b34c82ebe5" containerName="kserve-container"
Jul 28 16:17:29.046129 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.046110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.048251 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048228    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:17:29.048350 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8b634-predictor-serving-cert\""
Jul 28 16:17:29.048350 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:17:29.048676 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048659    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8b634-kube-rbac-proxy-sar-config\""
Jul 28 16:17:29.048755 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048739    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:17:29.048755 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:17:29.048992 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.048977    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:17:29.061926 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.061895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:17:29.174829 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.174794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.175018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.174837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.175018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.174885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khf5f\" (UniqueName: \"kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.175018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.174952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.175148 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.175020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276255 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khf5f\" (UniqueName: \"kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.276642 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:29.276448    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-serving-cert: secret "isvc-logger-raw-8b634-predictor-serving-cert" not found
Jul 28 16:17:29.276642 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:17:29.276525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls podName:1ec609b6-0402-46e9-ae0c-158e763604f1 nodeName:}" failed. No retries permitted until 2026-07-28 16:17:29.776505021 +0000 UTC m=+896.328103001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls") pod "isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1") : secret "isvc-logger-raw-8b634-predictor-serving-cert" not found
Jul 28 16:17:29.276803 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.276777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.277065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.277043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.277065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.277061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.286535 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.286507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khf5f\" (UniqueName: \"kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.781780 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.781745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.784114 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.784085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") pod \"isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:29.971788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:29.971753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:30.111343 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:30.111298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:17:30.116755 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:17:30.114961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec609b6_0402_46e9_ae0c_158e763604f1.slice/crio-40f0a5ef2b8ba2234fba6702a70d8ae1e22610f33a0d6e1429cfd0c659984a83 WatchSource:0}: Error finding container 40f0a5ef2b8ba2234fba6702a70d8ae1e22610f33a0d6e1429cfd0c659984a83: Status 404 returned error can't find the container with id 40f0a5ef2b8ba2234fba6702a70d8ae1e22610f33a0d6e1429cfd0c659984a83
Jul 28 16:17:30.951760 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:30.951724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerStarted","Data":"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"}
Jul 28 16:17:30.951996 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:30.951798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerStarted","Data":"40f0a5ef2b8ba2234fba6702a70d8ae1e22610f33a0d6e1429cfd0c659984a83"}
Jul 28 16:17:33.885066 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:33.885036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:17:33.885763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:33.885744    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:17:34.970060 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:34.970028    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerID="02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9" exitCode=0
Jul 28 16:17:34.970408 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:34.970104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerDied","Data":"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"}
Jul 28 16:17:35.975780 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.975740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerStarted","Data":"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"}
Jul 28 16:17:35.976296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.975817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerStarted","Data":"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"}
Jul 28 16:17:35.976296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.975887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerStarted","Data":"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"}
Jul 28 16:17:35.976296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.976159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:35.976296 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.976208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:35.977546 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.977520    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:17:35.998186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:35.998148    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podStartSLOduration=3.107749615 podStartE2EDuration="6.998133766s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.890384151s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:17:35.996318593 +0000 UTC m=+902.547916586" watchObservedRunningTime="2026-07-28 16:17:35.998133766 +0000 UTC m=+902.549731759"
Jul 28 16:17:36.984463 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:36.984430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:36.985332 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:36.985297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:17:36.985956 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:36.985931    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:17:37.987258 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:37.987208    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:17:37.987697 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:37.987676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:17:42.991872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:42.991821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:17:42.992395 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:42.992368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:17:42.992627 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:42.992601    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:17:52.992883 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:52.992823    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:17:52.993355 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:17:52.993279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:02.992748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:02.992707    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:18:02.993186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:02.993129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:12.993153 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:12.993102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:18:12.993637 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:12.993544    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:22.992867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:22.992820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:18:22.995406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:22.993313    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:32.993123 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:32.993081    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:18:32.993564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:32.993495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:42.993078 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:42.993040    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:18:42.993527 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:42.993467    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:18:52.993497 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:52.993458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:18:52.994010 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:18:52.993700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:04.316621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.316586    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:19:04.317090 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.316904    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" containerID="cri-o://2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b" gracePeriod=30
Jul 28 16:19:04.317090 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.316977    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" containerID="cri-o://87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f" gracePeriod=30
Jul 28 16:19:04.317212 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.317170    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" containerID="cri-o://d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345" gracePeriod=30
Jul 28 16:19:04.383048 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.383013    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:19:04.386578 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.386560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.388661 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.388646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-0a1ef-predictor-serving-cert\""
Jul 28 16:19:04.388777 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.388762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-0a1ef-kube-rbac-proxy-sar-config\""
Jul 28 16:19:04.397895 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.397874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:19:04.484900 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.484869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.485083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.484993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dxf\" (UniqueName: \"kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.485083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.485035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.485213 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.485102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.485213 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.485122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.585791 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.585705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dxf\" (UniqueName: \"kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.585791 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.585755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.585791 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.585794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.586103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.585821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.586103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.585873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.586231 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.586208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.586430 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.586409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.586502 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.586480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.588346 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.588323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.594953 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.594922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dxf\" (UniqueName: \"kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf\") pod \"isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.697019 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.696972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:04.833191 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:04.833159    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:19:04.836793 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:19:04.836717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62bc559_03a3_4fa9_8a92_bb4d08b8c3d3.slice/crio-62c568cb35a09ad42b978ae5ce1d65ee90172a307592f0bd3a51fae70cb1ddfe WatchSource:0}: Error finding container 62c568cb35a09ad42b978ae5ce1d65ee90172a307592f0bd3a51fae70cb1ddfe: Status 404 returned error can't find the container with id 62c568cb35a09ad42b978ae5ce1d65ee90172a307592f0bd3a51fae70cb1ddfe
Jul 28 16:19:05.342935 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:05.342886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerStarted","Data":"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"}
Jul 28 16:19:05.343392 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:05.343018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerStarted","Data":"62c568cb35a09ad42b978ae5ce1d65ee90172a307592f0bd3a51fae70cb1ddfe"}
Jul 28 16:19:05.345337 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:05.345313    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerID="87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f" exitCode=2
Jul 28 16:19:05.345434 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:05.345359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerDied","Data":"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"}
Jul 28 16:19:07.987966 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:07.987925    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:09.362942 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:09.362838    2579 generic.go:356] "Generic (PLEG): container finished" podID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerID="7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d" exitCode=0
Jul 28 16:19:09.362942 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:09.362912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerDied","Data":"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"}
Jul 28 16:19:09.365349 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:09.365328    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerID="2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b" exitCode=0
Jul 28 16:19:09.365453 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:09.365363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerDied","Data":"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"}
Jul 28 16:19:10.371422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:10.371389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerStarted","Data":"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"}
Jul 28 16:19:10.371798 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:10.371451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerStarted","Data":"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"}
Jul 28 16:19:10.371798 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:10.371644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:10.400808 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:10.400759    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podStartSLOduration=2.601453586 podStartE2EDuration="6.400744429s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799290843s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:19:10.39944623 +0000 UTC m=+996.951044222" watchObservedRunningTime="2026-07-28 16:19:10.400744429 +0000 UTC m=+996.952342421"
Jul 28 16:19:11.376179 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:11.376149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:11.377303 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:11.377274    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:12.385673 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:12.385632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:12.988422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:12.988377    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:12.992788 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:12.992759    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:19:12.993180 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:12.993152    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:19:17.390491 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:17.390461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:19:17.390998 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:17.390973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:17.987591 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:17.987550    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:17.987769 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:17.987669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:22.987634 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:22.987590    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:22.992991 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:22.992961    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:19:22.993315 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:22.993288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:19:27.391254 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:27.391209    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:27.987754 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:27.987702    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:32.987718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:32.987672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:19:32.993094 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:32.993065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:19:32.993208 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:32.993195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:32.993423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:32.993405    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:19:32.993491 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:32.993479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:34.470358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.470326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:34.480194 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.480167    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerID="d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345" exitCode=0
Jul 28 16:19:34.480322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.480217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerDied","Data":"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"}
Jul 28 16:19:34.480322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.480228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"
Jul 28 16:19:34.480322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.480271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc" event={"ID":"1ec609b6-0402-46e9-ae0c-158e763604f1","Type":"ContainerDied","Data":"40f0a5ef2b8ba2234fba6702a70d8ae1e22610f33a0d6e1429cfd0c659984a83"}
Jul 28 16:19:34.480322 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.480316    2579 scope.go:117] "RemoveContainer" containerID="d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"
Jul 28 16:19:34.488387 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.488367    2579 scope.go:117] "RemoveContainer" containerID="87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"
Jul 28 16:19:34.495649 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.495632    2579 scope.go:117] "RemoveContainer" containerID="2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"
Jul 28 16:19:34.503824 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.503805    2579 scope.go:117] "RemoveContainer" containerID="02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"
Jul 28 16:19:34.511876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.511837    2579 scope.go:117] "RemoveContainer" containerID="d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"
Jul 28 16:19:34.512155 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:19:34.512127    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345\": container with ID starting with d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345 not found: ID does not exist" containerID="d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"
Jul 28 16:19:34.512224 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512167    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345"} err="failed to get container status \"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345\": rpc error: code = NotFound desc = could not find container \"d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345\": container with ID starting with d9291c9b3cd94ad25853178b31da1a308a9e4f11cb23ea3b185badb4636e5345 not found: ID does not exist"
Jul 28 16:19:34.512224 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512190    2579 scope.go:117] "RemoveContainer" containerID="87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"
Jul 28 16:19:34.512511 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:19:34.512492    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f\": container with ID starting with 87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f not found: ID does not exist" containerID="87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"
Jul 28 16:19:34.512572 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512521    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f"} err="failed to get container status \"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f\": rpc error: code = NotFound desc = could not find container \"87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f\": container with ID starting with 87345c8e3089d8402a209e4cedc229f29f08b7b5f80f7b077f0b6a30982cc84f not found: ID does not exist"
Jul 28 16:19:34.512572 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512545    2579 scope.go:117] "RemoveContainer" containerID="2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"
Jul 28 16:19:34.512799 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:19:34.512781    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b\": container with ID starting with 2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b not found: ID does not exist" containerID="2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"
Jul 28 16:19:34.512896 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512807    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b"} err="failed to get container status \"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b\": rpc error: code = NotFound desc = could not find container \"2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b\": container with ID starting with 2a08009b8a373a76fe3153d8c567fb4bedd48fb6a222004dd49747ac891ca22b not found: ID does not exist"
Jul 28 16:19:34.512896 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.512828    2579 scope.go:117] "RemoveContainer" containerID="02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"
Jul 28 16:19:34.513102 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:19:34.513087    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9\": container with ID starting with 02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9 not found: ID does not exist" containerID="02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"
Jul 28 16:19:34.513144 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.513106    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9"} err="failed to get container status \"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9\": rpc error: code = NotFound desc = could not find container \"02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9\": container with ID starting with 02da95510fed215b7edce490e1c25eb64e1e91c7528ab3600814be69c17a7ef9 not found: ID does not exist"
Jul 28 16:19:34.646244 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khf5f\" (UniqueName: \"kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f\") pod \"1ec609b6-0402-46e9-ae0c-158e763604f1\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") "
Jul 28 16:19:34.646244 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") pod \"1ec609b6-0402-46e9-ae0c-158e763604f1\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") "
Jul 28 16:19:34.646473 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config\") pod \"1ec609b6-0402-46e9-ae0c-158e763604f1\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") "
Jul 28 16:19:34.646473 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location\") pod \"1ec609b6-0402-46e9-ae0c-158e763604f1\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") "
Jul 28 16:19:34.646473 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert\") pod \"1ec609b6-0402-46e9-ae0c-158e763604f1\" (UID: \"1ec609b6-0402-46e9-ae0c-158e763604f1\") "
Jul 28 16:19:34.646698 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646664    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ec609b6-0402-46e9-ae0c-158e763604f1" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:19:34.646698 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ec609b6-0402-46e9-ae0c-158e763604f1" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:19:34.646899 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.646716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ec609b6-0402-46e9-ae0c-158e763604f1" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:19:34.648320 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.648296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ec609b6-0402-46e9-ae0c-158e763604f1" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:19:34.648390 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.648339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f" (OuterVolumeSpecName: "kube-api-access-khf5f") pod "1ec609b6-0402-46e9-ae0c-158e763604f1" (UID: "1ec609b6-0402-46e9-ae0c-158e763604f1"). InnerVolumeSpecName "kube-api-access-khf5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:19:34.747210 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.747166    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:19:34.747210 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.747203    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec609b6-0402-46e9-ae0c-158e763604f1-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:19:34.747210 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.747214    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ec609b6-0402-46e9-ae0c-158e763604f1-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:19:34.747210 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.747224    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khf5f\" (UniqueName: \"kubernetes.io/projected/1ec609b6-0402-46e9-ae0c-158e763604f1-kube-api-access-khf5f\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:19:34.747474 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.747234    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec609b6-0402-46e9-ae0c-158e763604f1-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:19:34.803526 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.803491    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:19:34.806550 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:34.806528    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8b634-predictor-7c5f96c58b-x4xbc"]
Jul 28 16:19:36.016890 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:36.015681    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" path="/var/lib/kubelet/pods/1ec609b6-0402-46e9-ae0c-158e763604f1/volumes"
Jul 28 16:19:37.391138 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:37.391100    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:47.391580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:47.391539    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:19:57.391444 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:19:57.391397    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:07.391517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:07.391474    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:17.391028 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:17.390987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:22.010072 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:22.010024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:32.010026 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:32.009940    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:42.010686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:42.010637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:20:52.010812 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:20:52.010763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:02.010799 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:02.010752    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:12.010884 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:12.010825    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:22.010116 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:22.010076    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:28.010018 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:28.009970    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:38.010244 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:38.010203    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:48.014539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:48.014511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:21:54.628552 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:54.628515    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:21:54.629083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:54.628759    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" containerID="cri-o://096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443" gracePeriod=30
Jul 28 16:21:54.629083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:54.628802    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy" containerID="cri-o://6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6" gracePeriod=30
Jul 28 16:21:55.075107 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:55.075074    2579 generic.go:356] "Generic (PLEG): container finished" podID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerID="6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6" exitCode=2
Jul 28 16:21:55.075275 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:55.075146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerDied","Data":"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"}
Jul 28 16:21:57.386638 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:57.386596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 16:21:58.010145 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:58.010107    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:21:59.815424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815392    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:21:59.815762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="storage-initializer"
Jul 28 16:21:59.815762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815741    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="storage-initializer"
Jul 28 16:21:59.815762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy"
Jul 28 16:21:59.815762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815768    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815773    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815786    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815791    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="agent"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815872    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kube-rbac-proxy"
Jul 28 16:21:59.815937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.815880    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec609b6-0402-46e9-ae0c-158e763604f1" containerName="kserve-container"
Jul 28 16:21:59.818991 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.818976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.820871 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.820832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 16:21:59.821752 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.821735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 16:21:59.822864 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.822830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.822944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.822905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8z5z\" (UniqueName: \"kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.822944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.822925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.822944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.822944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.823067 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.823050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.828153 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.828132    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:21:59.832113 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.832095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.833212 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.833191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:21:59.834700 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.834673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 16:21:59.844911 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.844891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:21:59.923931 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.923896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.923944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.924100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.923979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndlm\" (UniqueName: \"kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.924100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.924100 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:21:59.924138    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8z5z\" (UniqueName: \"kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:21:59.924202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls podName:b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:00.424182678 +0000 UTC m=+1166.975780662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls") pod "isvc-canary-create-predictor-594f5f9bd5-hvvbz" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:21:59.924356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924749 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.924883 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.924838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:21:59.934916 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:21:59.934894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8z5z\" (UniqueName: \"kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:00.025477 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.025680 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.025680 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndlm\" (UniqueName: \"kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.025680 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.025680 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.026011 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.025989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.026169 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.026146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.026234 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.026215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.028202 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.028183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.035574 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.035546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndlm\" (UniqueName: \"kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm\") pod \"isvc-canary-create-v2-predictor-5b976d889b-cbsbk\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.144435 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.144341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:00.293210 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.293178    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:22:00.296184 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:22:00.296152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6415ca83_e451_431d_b61f_f681365d886d.slice/crio-b1f19812cb8eb951cf002f5029090684079cc378e714420f4e4f04e98b5a0eb6 WatchSource:0}: Error finding container b1f19812cb8eb951cf002f5029090684079cc378e714420f4e4f04e98b5a0eb6: Status 404 returned error can't find the container with id b1f19812cb8eb951cf002f5029090684079cc378e714420f4e4f04e98b5a0eb6
Jul 28 16:22:00.429237 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.429200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:00.431716 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.431690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-hvvbz\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:00.730373 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.730288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:00.864750 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:00.864583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:22:00.867157 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:22:00.867128    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b7f37a_05fa_47e6_ba12_c6b3615ef4b4.slice/crio-a9a3e5bf12e4e0e3fa62148fd66a254392a92d87988a1e0d5bdf95be06dd1560 WatchSource:0}: Error finding container a9a3e5bf12e4e0e3fa62148fd66a254392a92d87988a1e0d5bdf95be06dd1560: Status 404 returned error can't find the container with id a9a3e5bf12e4e0e3fa62148fd66a254392a92d87988a1e0d5bdf95be06dd1560
Jul 28 16:22:01.102640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:01.102543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerStarted","Data":"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"}
Jul 28 16:22:01.102640 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:01.102613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerStarted","Data":"b1f19812cb8eb951cf002f5029090684079cc378e714420f4e4f04e98b5a0eb6"}
Jul 28 16:22:01.104218 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:01.104193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerStarted","Data":"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"}
Jul 28 16:22:01.104367 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:01.104241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerStarted","Data":"a9a3e5bf12e4e0e3fa62148fd66a254392a92d87988a1e0d5bdf95be06dd1560"}
Jul 28 16:22:02.386746 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:02.386705    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 16:22:04.118716 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.118680    2579 generic.go:356] "Generic (PLEG): container finished" podID="6415ca83-e451-431d-b61f-f681365d886d" containerID="b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae" exitCode=0
Jul 28 16:22:04.119128 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.118735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerDied","Data":"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"}
Jul 28 16:22:04.597287 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.597262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:22:04.668124 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config\") pod \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") "
Jul 28 16:22:04.668300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert\") pod \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") "
Jul 28 16:22:04.668300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64dxf\" (UniqueName: \"kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf\") pod \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") "
Jul 28 16:22:04.668300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location\") pod \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") "
Jul 28 16:22:04.668300 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls\") pod \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\" (UID: \"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3\") "
Jul 28 16:22:04.668523 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" (UID: "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:04.668523 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" (UID: "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:04.668613 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.668526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" (UID: "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:22:04.670276 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.670252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf" (OuterVolumeSpecName: "kube-api-access-64dxf") pod "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" (UID: "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3"). InnerVolumeSpecName "kube-api-access-64dxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:04.670360 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.670306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" (UID: "e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:04.769939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.769862    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:22:04.769939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.769889    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:22:04.769939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.769900    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dxf\" (UniqueName: \"kubernetes.io/projected/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kube-api-access-64dxf\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:22:04.769939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.769910    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:22:04.769939 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:04.769920    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:22:05.124645 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.124559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerStarted","Data":"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"}
Jul 28 16:22:05.124645 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.124626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerStarted","Data":"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"}
Jul 28 16:22:05.125103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.124893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:05.126358 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.126336    2579 generic.go:356] "Generic (PLEG): container finished" podID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerID="096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443" exitCode=0
Jul 28 16:22:05.126444 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.126387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"
Jul 28 16:22:05.126444 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.126396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerDied","Data":"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"}
Jul 28 16:22:05.126547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.126441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k" event={"ID":"e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3","Type":"ContainerDied","Data":"62c568cb35a09ad42b978ae5ce1d65ee90172a307592f0bd3a51fae70cb1ddfe"}
Jul 28 16:22:05.126547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.126473    2579 scope.go:117] "RemoveContainer" containerID="6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"
Jul 28 16:22:05.127976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.127953    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerID="4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b" exitCode=0
Jul 28 16:22:05.128065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.128007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerDied","Data":"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"}
Jul 28 16:22:05.135522 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.135499    2579 scope.go:117] "RemoveContainer" containerID="096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"
Jul 28 16:22:05.143797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.143776    2579 scope.go:117] "RemoveContainer" containerID="7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"
Jul 28 16:22:05.149276 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.149233    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podStartSLOduration=2.531340486 podStartE2EDuration="6.149218884s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.617878398s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:05.147324859 +0000 UTC m=+1171.698922873" watchObservedRunningTime="2026-07-28 16:22:05.149218884 +0000 UTC m=+1171.700816877"
Jul 28 16:22:05.157392 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.157371    2579 scope.go:117] "RemoveContainer" containerID="6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"
Jul 28 16:22:05.157681 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:22:05.157663    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6\": container with ID starting with 6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6 not found: ID does not exist" containerID="6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"
Jul 28 16:22:05.157727 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.157692    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6"} err="failed to get container status \"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6\": rpc error: code = NotFound desc = could not find container \"6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6\": container with ID starting with 6428ce5c333c78f774e45eb727015863635a26b541d34f903ad431237d2488a6 not found: ID does not exist"
Jul 28 16:22:05.157727 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.157722    2579 scope.go:117] "RemoveContainer" containerID="096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"
Jul 28 16:22:05.157972 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:22:05.157955    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443\": container with ID starting with 096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443 not found: ID does not exist" containerID="096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"
Jul 28 16:22:05.158043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.157983    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443"} err="failed to get container status \"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443\": rpc error: code = NotFound desc = could not find container \"096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443\": container with ID starting with 096e1fbfcdf8cef19997a5eb487a68bbecd6750b4c2312c2985c920372b2a443 not found: ID does not exist"
Jul 28 16:22:05.158043 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.158005    2579 scope.go:117] "RemoveContainer" containerID="7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"
Jul 28 16:22:05.158258 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:22:05.158242    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d\": container with ID starting with 7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d not found: ID does not exist" containerID="7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"
Jul 28 16:22:05.158295 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.158262    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d"} err="failed to get container status \"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d\": rpc error: code = NotFound desc = could not find container \"7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d\": container with ID starting with 7626c9fa9ee2b1ff481fc2bf5cca6c47f798962f87c8d951a48668033543cc0d not found: ID does not exist"
Jul 28 16:22:05.166976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.166955    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:22:05.172822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:05.172799    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-0a1ef-predictor-5d7b7b4d54-frf2k"]
Jul 28 16:22:06.014297 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.014264    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" path="/var/lib/kubelet/pods/e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3/volumes"
Jul 28 16:22:06.135686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.135655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerStarted","Data":"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"}
Jul 28 16:22:06.136078 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.135712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerStarted","Data":"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"}
Jul 28 16:22:06.136078 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.135952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:06.136206 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.136184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:06.137203 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.137181    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:06.165415 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:06.165368    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podStartSLOduration=3.551056483 podStartE2EDuration="7.165354039s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.614297556s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:06.162316277 +0000 UTC m=+1172.713914271" watchObservedRunningTime="2026-07-28 16:22:06.165354039 +0000 UTC m=+1172.716952031"
Jul 28 16:22:07.146537 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:07.146508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:07.146537 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:07.146525    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:07.147702 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:07.147678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:08.150817 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:08.150777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:12.150519 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:12.150492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:22:12.151026 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:12.151002    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:13.155063 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:13.155036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:22:13.155569 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:13.155547    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:22.151229 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:22.151192    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:23.156506 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:23.156465    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:32.151539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:32.151500    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:33.155531 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:33.155495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:33.913668 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:33.913636    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:22:33.914800 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:33.914780    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:22:42.151128 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:42.151087    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:43.156273 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:43.156234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:22:52.151305 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:52.151268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:53.156465 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:22:53.156426    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:23:02.151591 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:02.151556    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:23:03.156083 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:03.156044    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:23:12.153828 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:12.153779    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:23:13.010307 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:13.010266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:23:13.156103 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:13.156064    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:23:23.011097 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:23.010994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:23:23.155993 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:23.155966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:23:29.976209 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.976123    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:23:29.976618 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.976374    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" containerID="cri-o://06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871" gracePeriod=30
Jul 28 16:23:29.976618 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.976393    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kube-rbac-proxy" containerID="cri-o://f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e" gracePeriod=30
Jul 28 16:23:29.989220 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.989193    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:23:29.989412 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.989394    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" containerID="cri-o://bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516" gracePeriod=30
Jul 28 16:23:29.989501 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:29.989464    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kube-rbac-proxy" containerID="cri-o://f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88" gracePeriod=30
Jul 28 16:23:30.519867 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:30.519817    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerID="f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e" exitCode=2
Jul 28 16:23:30.520040 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:30.519897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerDied","Data":"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"}
Jul 28 16:23:30.521762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:30.521741    2579 generic.go:356] "Generic (PLEG): container finished" podID="6415ca83-e451-431d-b61f-f681365d886d" containerID="f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88" exitCode=2
Jul 28 16:23:30.521880 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:30.521813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerDied","Data":"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"}
Jul 28 16:23:32.146861 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:32.146793    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 16:23:33.011113 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:33.011078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:23:33.151770 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:33.151730    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 16:23:33.156162 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:33.156137    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:23:34.952992 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:34.952971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:23:34.955793 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:34.955775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:23:35.040361 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config\") pod \"6415ca83-e451-431d-b61f-f681365d886d\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") "
Jul 28 16:23:35.040361 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert\") pod \"6415ca83-e451-431d-b61f-f681365d886d\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") pod \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert\") pod \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nndlm\" (UniqueName: \"kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm\") pod \"6415ca83-e451-431d-b61f-f681365d886d\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location\") pod \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config\") pod \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") "
Jul 28 16:23:35.040543 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040541    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8z5z\" (UniqueName: \"kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z\") pod \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\" (UID: \"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4\") "
Jul 28 16:23:35.040752 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls\") pod \"6415ca83-e451-431d-b61f-f681365d886d\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") "
Jul 28 16:23:35.040752 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location\") pod \"6415ca83-e451-431d-b61f-f681365d886d\" (UID: \"6415ca83-e451-431d-b61f-f681365d886d\") "
Jul 28 16:23:35.040820 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:35.040969 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.040953    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.041133 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.041110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6415ca83-e451-431d-b61f-f681365d886d" (UID: "6415ca83-e451-431d-b61f-f681365d886d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:23:35.041327 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.041308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6415ca83-e451-431d-b61f-f681365d886d" (UID: "6415ca83-e451-431d-b61f-f681365d886d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:35.042340 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.041936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:35.042453 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.042018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6415ca83-e451-431d-b61f-f681365d886d" (UID: "6415ca83-e451-431d-b61f-f681365d886d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:35.042607 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.042572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:23:35.043828 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.043794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm" (OuterVolumeSpecName: "kube-api-access-nndlm") pod "6415ca83-e451-431d-b61f-f681365d886d" (UID: "6415ca83-e451-431d-b61f-f681365d886d"). InnerVolumeSpecName "kube-api-access-nndlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:23:35.044272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.044247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:23:35.045104 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.045080    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z" (OuterVolumeSpecName: "kube-api-access-f8z5z") pod "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" (UID: "b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4"). InnerVolumeSpecName "kube-api-access-f8z5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:23:35.045195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.045086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6415ca83-e451-431d-b61f-f681365d886d" (UID: "6415ca83-e451-431d-b61f-f681365d886d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:23:35.128908 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.128872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:23:35.129263 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="storage-initializer"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="storage-initializer"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129280    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129288    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129301    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129310    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="storage-initializer"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129341    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="storage-initializer"
Jul 28 16:23:35.129356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129351    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129359    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129379    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129388    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129400    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129408    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129418    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="storage-initializer"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129427    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="storage-initializer"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129446    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129532    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kserve-container"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129543    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kserve-container"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129554    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kserve-container"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129566    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62bc559-03a3-4fa9-8a92-bb4d08b8c3d3" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerName="kube-rbac-proxy"
Jul 28 16:23:35.129748 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.129586    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6415ca83-e451-431d-b61f-f681365d886d" containerName="kube-rbac-proxy"
Jul 28 16:23:35.134277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.134254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.136645 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.136623    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 16:23:35.136762 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.136664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 16:23:35.141548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141486    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6415ca83-e451-431d-b61f-f681365d886d-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141515    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6415ca83-e451-431d-b61f-f681365d886d-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141529    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141545    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6415ca83-e451-431d-b61f-f681365d886d-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141558    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndlm\" (UniqueName: \"kubernetes.io/projected/6415ca83-e451-431d-b61f-f681365d886d-kube-api-access-nndlm\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141585    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141598    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141611    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8z5z\" (UniqueName: \"kubernetes.io/projected/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4-kube-api-access-f8z5z\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:23:35.141823 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.141642    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:23:35.145672 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.145651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.145802 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.145785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:23:35.147963 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.147944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 16:23:35.160638 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.160608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:23:35.242214 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.242214 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhmp\" (UniqueName: \"kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.242423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.242423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.242423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.242423 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.242560 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw67c\" (UniqueName: \"kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.242560 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.242560 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.242560 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.242513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.343980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.343884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.343980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.343930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhmp\" (UniqueName: \"kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.343980 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.343955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw67c\" (UniqueName: \"kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.344598 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.344598 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.344813 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.344784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.345047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.345011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.345170 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.345056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.345264 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.345240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.346944 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.346920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.347023 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.346920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.361650 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.361629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw67c\" (UniqueName: \"kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-hj5mn\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.363215 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.363199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhmp\" (UniqueName: \"kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-qkh5t\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.446314 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.446269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:35.456551 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.456518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:35.546484 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.546457    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" containerID="06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871" exitCode=0
Jul 28 16:23:35.546623 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.546535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"
Jul 28 16:23:35.546623 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.546543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerDied","Data":"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"}
Jul 28 16:23:35.546811 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.546627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz" event={"ID":"b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4","Type":"ContainerDied","Data":"a9a3e5bf12e4e0e3fa62148fd66a254392a92d87988a1e0d5bdf95be06dd1560"}
Jul 28 16:23:35.546811 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.546672    2579 scope.go:117] "RemoveContainer" containerID="f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"
Jul 28 16:23:35.549062 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.548979    2579 generic.go:356] "Generic (PLEG): container finished" podID="6415ca83-e451-431d-b61f-f681365d886d" containerID="bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516" exitCode=0
Jul 28 16:23:35.549062 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.549042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"
Jul 28 16:23:35.549243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.549085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerDied","Data":"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"}
Jul 28 16:23:35.549243 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.549137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk" event={"ID":"6415ca83-e451-431d-b61f-f681365d886d","Type":"ContainerDied","Data":"b1f19812cb8eb951cf002f5029090684079cc378e714420f4e4f04e98b5a0eb6"}
Jul 28 16:23:35.560610 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.558009    2579 scope.go:117] "RemoveContainer" containerID="06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"
Jul 28 16:23:35.575960 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.575936    2579 scope.go:117] "RemoveContainer" containerID="4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"
Jul 28 16:23:35.584780 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.584756    2579 scope.go:117] "RemoveContainer" containerID="f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"
Jul 28 16:23:35.585125 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.585098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e\": container with ID starting with f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e not found: ID does not exist" containerID="f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"
Jul 28 16:23:35.585200 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585134    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e"} err="failed to get container status \"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e\": rpc error: code = NotFound desc = could not find container \"f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e\": container with ID starting with f696b2010b4121e78225f1daf6ba0e7e83da86a5909fb4f0509393e20a12b34e not found: ID does not exist"
Jul 28 16:23:35.585200 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585153    2579 scope.go:117] "RemoveContainer" containerID="06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"
Jul 28 16:23:35.585449 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.585432    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871\": container with ID starting with 06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871 not found: ID does not exist" containerID="06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"
Jul 28 16:23:35.585501 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585456    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871"} err="failed to get container status \"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871\": rpc error: code = NotFound desc = could not find container \"06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871\": container with ID starting with 06b13b85075bf43ca2f5f83a3f2d334779ae11a433f885fed18c39b462eff871 not found: ID does not exist"
Jul 28 16:23:35.585501 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585472    2579 scope.go:117] "RemoveContainer" containerID="4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"
Jul 28 16:23:35.585771 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.585748    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b\": container with ID starting with 4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b not found: ID does not exist" containerID="4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"
Jul 28 16:23:35.585872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585785    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b"} err="failed to get container status \"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b\": rpc error: code = NotFound desc = could not find container \"4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b\": container with ID starting with 4ff97bea102e833c176ec9cab03f5d493120c4785479f6e8236bd1337d89310b not found: ID does not exist"
Jul 28 16:23:35.585872 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.585801    2579 scope.go:117] "RemoveContainer" containerID="f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"
Jul 28 16:23:35.590265 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.590242    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:23:35.596461 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.596440    2579 scope.go:117] "RemoveContainer" containerID="bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"
Jul 28 16:23:35.604250 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.604236    2579 scope.go:117] "RemoveContainer" containerID="b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"
Jul 28 16:23:35.609384 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.609362    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-cbsbk"]
Jul 28 16:23:35.612669 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.612648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:23:35.613169 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.613155    2579 scope.go:117] "RemoveContainer" containerID="f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"
Jul 28 16:23:35.613470 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.613434    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88\": container with ID starting with f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88 not found: ID does not exist" containerID="f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"
Jul 28 16:23:35.613470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.613464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88"} err="failed to get container status \"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88\": rpc error: code = NotFound desc = could not find container \"f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88\": container with ID starting with f19bc1703012434928a2f8a294823f3a5149a7d8a7fa25b0a3c151646808ff88 not found: ID does not exist"
Jul 28 16:23:35.613686 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.613481    2579 scope.go:117] "RemoveContainer" containerID="bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"
Jul 28 16:23:35.613744 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.613724    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516\": container with ID starting with bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516 not found: ID does not exist" containerID="bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"
Jul 28 16:23:35.613783 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.613753    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516"} err="failed to get container status \"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516\": rpc error: code = NotFound desc = could not find container \"bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516\": container with ID starting with bb2de0b9df4e84d748300de3e604e7f6866cb2ea4a52a87590fafebd52c6c516 not found: ID does not exist"
Jul 28 16:23:35.613783 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.613772    2579 scope.go:117] "RemoveContainer" containerID="b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"
Jul 28 16:23:35.614066 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:23:35.614029    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae\": container with ID starting with b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae not found: ID does not exist" containerID="b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"
Jul 28 16:23:35.614066 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.614046    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae"} err="failed to get container status \"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae\": rpc error: code = NotFound desc = could not find container \"b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae\": container with ID starting with b387db2d0f686831c967ed899b7f3b306c69e6df8546761532f2ec9bcf9b3eae not found: ID does not exist"
Jul 28 16:23:35.616304 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:23:35.616281    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c47e348_2ad1_4ad4_b517_e18574dcaeb2.slice/crio-79078f45dc333d225cbd662d202c66968e9f12dd42f3bb4fc6423b6d6b11b767 WatchSource:0}: Error finding container 79078f45dc333d225cbd662d202c66968e9f12dd42f3bb4fc6423b6d6b11b767: Status 404 returned error can't find the container with id 79078f45dc333d225cbd662d202c66968e9f12dd42f3bb4fc6423b6d6b11b767
Jul 28 16:23:35.643779 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.643754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:23:35.649406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.649363    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-hvvbz"]
Jul 28 16:23:35.655466 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:35.655444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:23:35.658890 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:23:35.658866    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca384f85_fcaf_4af5_af08_c2b6b816c656.slice/crio-98188b050082151b017c9a428f41c51379c14eb8dc0ff5899637c3c1a8b80ccb WatchSource:0}: Error finding container 98188b050082151b017c9a428f41c51379c14eb8dc0ff5899637c3c1a8b80ccb: Status 404 returned error can't find the container with id 98188b050082151b017c9a428f41c51379c14eb8dc0ff5899637c3c1a8b80ccb
Jul 28 16:23:36.016047 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.016014    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6415ca83-e451-431d-b61f-f681365d886d" path="/var/lib/kubelet/pods/6415ca83-e451-431d-b61f-f681365d886d/volumes"
Jul 28 16:23:36.016501 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.016488    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4" path="/var/lib/kubelet/pods/b0b7f37a-05fa-47e6-ba12-c6b3615ef4b4/volumes"
Jul 28 16:23:36.556492 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.556459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerStarted","Data":"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"}
Jul 28 16:23:36.556658 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.556516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerStarted","Data":"79078f45dc333d225cbd662d202c66968e9f12dd42f3bb4fc6423b6d6b11b767"}
Jul 28 16:23:36.557812 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.557786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerStarted","Data":"ced679db7a188e23ce06b85c4c02c935dd5c492da2a5942678a2bdf6dc51a201"}
Jul 28 16:23:36.557937 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:36.557841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerStarted","Data":"98188b050082151b017c9a428f41c51379c14eb8dc0ff5899637c3c1a8b80ccb"}
Jul 28 16:23:39.574837 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:39.574749    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerID="bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9" exitCode=0
Jul 28 16:23:39.574837 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:39.574825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerDied","Data":"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"}
Jul 28 16:23:39.576430 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:39.576408    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerID="ced679db7a188e23ce06b85c4c02c935dd5c492da2a5942678a2bdf6dc51a201" exitCode=0
Jul 28 16:23:39.576527 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:39.576477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerDied","Data":"ced679db7a188e23ce06b85c4c02c935dd5c492da2a5942678a2bdf6dc51a201"}
Jul 28 16:23:40.583272 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.583237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerStarted","Data":"2e79de8fcd7fdc6b167959086cae903431e90a9b2ed0b11c1f2fa20493155c67"}
Jul 28 16:23:40.583729 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.583298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerStarted","Data":"84f4fbbab928f19504405a967a02805e0d7bfb89d0a00d8a61e174c7504f2753"}
Jul 28 16:23:40.583729 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.583608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:40.583877 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.583747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:40.584683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.584655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:23:40.585297 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.585271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerStarted","Data":"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"}
Jul 28 16:23:40.585356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.585315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerStarted","Data":"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"}
Jul 28 16:23:40.585578 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.585557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:40.585685 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.585598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:40.586622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.586592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:23:40.612336 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.612293    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podStartSLOduration=1.946168531 podStartE2EDuration="5.612281723s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.666113192s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:40.610680093 +0000 UTC m=+1267.162278088" watchObservedRunningTime="2026-07-28 16:23:40.612281723 +0000 UTC m=+1267.163879716"
Jul 28 16:23:40.640390 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:40.640347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podStartSLOduration=2.119715344 podStartE2EDuration="5.640333212s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.520617868s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:40.638805875 +0000 UTC m=+1267.190403880" watchObservedRunningTime="2026-07-28 16:23:40.640333212 +0000 UTC m=+1267.191931205"
Jul 28 16:23:41.589728 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:41.589685    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:23:41.590195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:41.589760    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:23:46.595170 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:46.595136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:23:46.595976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:46.595943    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:23:46.595976 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:46.595968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:23:46.596436 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:46.596411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:23:56.596785 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:56.596746    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:23:56.597195 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:23:56.596747    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:06.596332 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:06.596290    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:24:06.596743 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:06.596431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:16.596177 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:16.596141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:24:16.596571 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:16.596357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:26.596452 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:26.596409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:26.596920 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:26.596410    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:24:36.596339 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:36.596294    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:24:36.596794 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:36.596435    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:46.596038 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:46.595994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:24:46.596420 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:46.596369    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:24:56.597038 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:56.596962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:24:56.597406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:24:56.597052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:25:05.224060 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:05.224017    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:25:05.224580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:05.224316    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" containerID="cri-o://e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab" gracePeriod=30
Jul 28 16:25:05.224580 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:05.224366    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kube-rbac-proxy" containerID="cri-o://af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850" gracePeriod=30
Jul 28 16:25:05.970257 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:05.970220    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerID="af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850" exitCode=2
Jul 28 16:25:05.970424 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:05.970268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerDied","Data":"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"}
Jul 28 16:25:06.590310 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:06.590263    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 16:25:06.596886 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:06.596835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:25:09.973632 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.973603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:25:09.989057 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.989029    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerID="e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab" exitCode=0
Jul 28 16:25:09.989152 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.989078    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"
Jul 28 16:25:09.989152 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.989096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerDied","Data":"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"}
Jul 28 16:25:09.989231 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.989167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn" event={"ID":"4c47e348-2ad1-4ad4-b517-e18574dcaeb2","Type":"ContainerDied","Data":"79078f45dc333d225cbd662d202c66968e9f12dd42f3bb4fc6423b6d6b11b767"}
Jul 28 16:25:09.989231 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.989194    2579 scope.go:117] "RemoveContainer" containerID="af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"
Jul 28 16:25:09.997422 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:09.997402    2579 scope.go:117] "RemoveContainer" containerID="e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"
Jul 28 16:25:10.005870 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.005831    2579 scope.go:117] "RemoveContainer" containerID="bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"
Jul 28 16:25:10.012987 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.012970    2579 scope.go:117] "RemoveContainer" containerID="af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"
Jul 28 16:25:10.013237 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:25:10.013216    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850\": container with ID starting with af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850 not found: ID does not exist" containerID="af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"
Jul 28 16:25:10.013330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.013248    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850"} err="failed to get container status \"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850\": rpc error: code = NotFound desc = could not find container \"af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850\": container with ID starting with af96981a9132dc113c23ac0c889652eabb1b9b90dbd141febe40571b54251850 not found: ID does not exist"
Jul 28 16:25:10.013330 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.013272    2579 scope.go:117] "RemoveContainer" containerID="e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"
Jul 28 16:25:10.013551 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:25:10.013528    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab\": container with ID starting with e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab not found: ID does not exist" containerID="e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"
Jul 28 16:25:10.013600 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.013563    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab"} err="failed to get container status \"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab\": rpc error: code = NotFound desc = could not find container \"e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab\": container with ID starting with e56c086d66c961afb8ecf449d68847edf63cf82cf02b943e55a827c8953525ab not found: ID does not exist"
Jul 28 16:25:10.013600 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.013586    2579 scope.go:117] "RemoveContainer" containerID="bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"
Jul 28 16:25:10.013879 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:25:10.013857    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9\": container with ID starting with bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9 not found: ID does not exist" containerID="bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"
Jul 28 16:25:10.013971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.013883    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9"} err="failed to get container status \"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9\": rpc error: code = NotFound desc = could not find container \"bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9\": container with ID starting with bcb820b958521b8a9dda07f05ab961faaf0638046a6b772e4c6b66f72be00ac9 not found: ID does not exist"
Jul 28 16:25:10.086328 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location\") pod \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") "
Jul 28 16:25:10.086500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert\") pod \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") "
Jul 28 16:25:10.086500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls\") pod \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") "
Jul 28 16:25:10.086500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw67c\" (UniqueName: \"kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c\") pod \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") "
Jul 28 16:25:10.086500 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config\") pod \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\" (UID: \"4c47e348-2ad1-4ad4-b517-e18574dcaeb2\") "
Jul 28 16:25:10.086718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c47e348-2ad1-4ad4-b517-e18574dcaeb2" (UID: "4c47e348-2ad1-4ad4-b517-e18574dcaeb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:25:10.086776 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086744    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c47e348-2ad1-4ad4-b517-e18574dcaeb2" (UID: "4c47e348-2ad1-4ad4-b517-e18574dcaeb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:25:10.086958 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.086934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c47e348-2ad1-4ad4-b517-e18574dcaeb2" (UID: "4c47e348-2ad1-4ad4-b517-e18574dcaeb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:25:10.088593 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.088569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c" (OuterVolumeSpecName: "kube-api-access-zw67c") pod "4c47e348-2ad1-4ad4-b517-e18574dcaeb2" (UID: "4c47e348-2ad1-4ad4-b517-e18574dcaeb2"). InnerVolumeSpecName "kube-api-access-zw67c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:25:10.088653 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.088593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c47e348-2ad1-4ad4-b517-e18574dcaeb2" (UID: "4c47e348-2ad1-4ad4-b517-e18574dcaeb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:25:10.187119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.187086    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:25:10.187119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.187112    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw67c\" (UniqueName: \"kubernetes.io/projected/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-api-access-zw67c\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:25:10.187119 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.187122    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:25:10.187321 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.187131    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:25:10.187321 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.187141    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c47e348-2ad1-4ad4-b517-e18574dcaeb2-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:25:10.316777 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.316741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:25:10.321613 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:10.321589    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-hj5mn"]
Jul 28 16:25:12.014745 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:25:12.014707    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" path="/var/lib/kubelet/pods/4c47e348-2ad1-4ad4-b517-e18574dcaeb2/volumes"
Jul 28 16:26:55.730102 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:55.730027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:26:55.730583 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:55.730286    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" containerID="cri-o://84f4fbbab928f19504405a967a02805e0d7bfb89d0a00d8a61e174c7504f2753" gracePeriod=30
Jul 28 16:26:55.730583 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:55.730321    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kube-rbac-proxy" containerID="cri-o://2e79de8fcd7fdc6b167959086cae903431e90a9b2ed0b11c1f2fa20493155c67" gracePeriod=30
Jul 28 16:26:56.430799 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:56.430768    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerID="2e79de8fcd7fdc6b167959086cae903431e90a9b2ed0b11c1f2fa20493155c67" exitCode=2
Jul 28 16:26:56.430986 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:56.430831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerDied","Data":"2e79de8fcd7fdc6b167959086cae903431e90a9b2ed0b11c1f2fa20493155c67"}
Jul 28 16:26:56.590781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:56.590741    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 16:26:56.596143 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:26:56.596120    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:27:00.447786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.447752    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerID="84f4fbbab928f19504405a967a02805e0d7bfb89d0a00d8a61e174c7504f2753" exitCode=0
Jul 28 16:27:00.448146 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.447787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerDied","Data":"84f4fbbab928f19504405a967a02805e0d7bfb89d0a00d8a61e174c7504f2753"}
Jul 28 16:27:00.478073 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.478042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:27:00.554732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.554640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert\") pod \"ca384f85-fcaf-4af5-af08-c2b6b816c656\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") "
Jul 28 16:27:00.554732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.554692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvhmp\" (UniqueName: \"kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp\") pod \"ca384f85-fcaf-4af5-af08-c2b6b816c656\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") "
Jul 28 16:27:00.554732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.554712    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls\") pod \"ca384f85-fcaf-4af5-af08-c2b6b816c656\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") "
Jul 28 16:27:00.554732 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.554737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location\") pod \"ca384f85-fcaf-4af5-af08-c2b6b816c656\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") "
Jul 28 16:27:00.555105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.554761    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config\") pod \"ca384f85-fcaf-4af5-af08-c2b6b816c656\" (UID: \"ca384f85-fcaf-4af5-af08-c2b6b816c656\") "
Jul 28 16:27:00.555105 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.555053    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca384f85-fcaf-4af5-af08-c2b6b816c656" (UID: "ca384f85-fcaf-4af5-af08-c2b6b816c656"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:27:00.555197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.555114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca384f85-fcaf-4af5-af08-c2b6b816c656" (UID: "ca384f85-fcaf-4af5-af08-c2b6b816c656"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:27:00.555197 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.555160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca384f85-fcaf-4af5-af08-c2b6b816c656" (UID: "ca384f85-fcaf-4af5-af08-c2b6b816c656"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:27:00.556981 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.556950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp" (OuterVolumeSpecName: "kube-api-access-vvhmp") pod "ca384f85-fcaf-4af5-af08-c2b6b816c656" (UID: "ca384f85-fcaf-4af5-af08-c2b6b816c656"). InnerVolumeSpecName "kube-api-access-vvhmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:27:00.557077 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.556990    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca384f85-fcaf-4af5-af08-c2b6b816c656" (UID: "ca384f85-fcaf-4af5-af08-c2b6b816c656"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:27:00.655490 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.655454    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhmp\" (UniqueName: \"kubernetes.io/projected/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-api-access-vvhmp\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:27:00.655718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.655494    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca384f85-fcaf-4af5-af08-c2b6b816c656-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:27:00.655718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.655511    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca384f85-fcaf-4af5-af08-c2b6b816c656-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:27:00.655718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.655524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:27:00.655718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:00.655537    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca384f85-fcaf-4af5-af08-c2b6b816c656-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:27:01.453475 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.453437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t" event={"ID":"ca384f85-fcaf-4af5-af08-c2b6b816c656","Type":"ContainerDied","Data":"98188b050082151b017c9a428f41c51379c14eb8dc0ff5899637c3c1a8b80ccb"}
Jul 28 16:27:01.453475 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.453480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"
Jul 28 16:27:01.454051 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.453502    2579 scope.go:117] "RemoveContainer" containerID="2e79de8fcd7fdc6b167959086cae903431e90a9b2ed0b11c1f2fa20493155c67"
Jul 28 16:27:01.462798 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.462779    2579 scope.go:117] "RemoveContainer" containerID="84f4fbbab928f19504405a967a02805e0d7bfb89d0a00d8a61e174c7504f2753"
Jul 28 16:27:01.470026 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.470008    2579 scope.go:117] "RemoveContainer" containerID="ced679db7a188e23ce06b85c4c02c935dd5c492da2a5942678a2bdf6dc51a201"
Jul 28 16:27:01.477608 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.477588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:27:01.484967 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:01.484946    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-qkh5t"]
Jul 28 16:27:02.014193 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:02.014164    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" path="/var/lib/kubelet/pods/ca384f85-fcaf-4af5-af08-c2b6b816c656/volumes"
Jul 28 16:27:33.939501 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:33.939468    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:27:33.941538 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:27:33.941515    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:28:30.861097 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861060    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861551    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="storage-initializer"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861566    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="storage-initializer"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861574    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861583    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861591    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kube-rbac-proxy"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861597    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kube-rbac-proxy"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861607    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="storage-initializer"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861612    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="storage-initializer"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861627    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861632    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kube-rbac-proxy"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861637    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kube-rbac-proxy"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861698    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kserve-container"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861703    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca384f85-fcaf-4af5-af08-c2b6b816c656" containerName="kube-rbac-proxy"
Jul 28 16:28:30.863670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.861711    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c47e348-2ad1-4ad4-b517-e18574dcaeb2" containerName="kube-rbac-proxy"
Jul 28 16:28:30.864737 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.864721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:30.867963 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.867941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 16:28:30.868099 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.867984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 16:28:30.868533 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.868512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:28:30.868632 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.868613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:28:30.868632 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.868628    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:28:30.869634 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.869621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:28:30.870929 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.870913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:28:30.879565 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.879546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:28:30.941961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.941921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:30.942132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.941977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vzf\" (UniqueName: \"kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:30.942132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.942056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:30.942132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.942100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:30.942132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:30.942121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043130 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vzf\" (UniqueName: \"kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.043977 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.044025 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.043953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.045836 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.045813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.053286 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.053262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vzf\" (UniqueName: \"kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7hbk5\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.175319 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.175242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:31.310403 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.310366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:28:31.313490 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:28:31.313454    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3d027f_8b99_40f7_9111_1ededbf24825.slice/crio-ee4627fb9e687e4b9d44eaf525476a2ce12bac5595c31098224501393e82b626 WatchSource:0}: Error finding container ee4627fb9e687e4b9d44eaf525476a2ce12bac5595c31098224501393e82b626: Status 404 returned error can't find the container with id ee4627fb9e687e4b9d44eaf525476a2ce12bac5595c31098224501393e82b626
Jul 28 16:28:31.814108 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.814078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerStarted","Data":"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"}
Jul 28 16:28:31.814274 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:31.814132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerStarted","Data":"ee4627fb9e687e4b9d44eaf525476a2ce12bac5595c31098224501393e82b626"}
Jul 28 16:28:35.830957 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:35.830921    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerID="328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0" exitCode=0
Jul 28 16:28:35.831391 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:35.830994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerDied","Data":"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"}
Jul 28 16:28:36.837286 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.837252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerStarted","Data":"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"}
Jul 28 16:28:36.837706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.837312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerStarted","Data":"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"}
Jul 28 16:28:36.837706 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.837565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:36.837905 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.837734    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:36.838910 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.838880    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:28:36.860834 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:36.860781    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podStartSLOduration=3.240068775 podStartE2EDuration="6.860762827s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.620694052s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:28:36.859214168 +0000 UTC m=+1563.410812159" watchObservedRunningTime="2026-07-28 16:28:36.860762827 +0000 UTC m=+1563.412360820"
Jul 28 16:28:37.843040 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:37.842999    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:28:42.847667 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:42.847638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:28:42.848156 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:42.848127    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:28:52.848114 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:28:52.848070    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:02.848283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:02.848243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:12.848811 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:12.848759    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:22.849032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:22.848995    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:32.848822 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:32.848731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:42.848214 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:42.848172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:29:52.848900 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:29:52.848868    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:30:01.024879 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:01.024832    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:30:01.025384 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:01.025093    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" containerID="cri-o://1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e" gracePeriod=30
Jul 28 16:30:01.025384 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:01.025148    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kube-rbac-proxy" containerID="cri-o://69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981" gracePeriod=30
Jul 28 16:30:01.173550 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:01.173519    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerID="69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981" exitCode=2
Jul 28 16:30:01.173712 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:01.173590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerDied","Data":"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"}
Jul 28 16:30:02.844345 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:02.844303    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 16:30:02.848904 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:02.848875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:30:05.764411 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.764387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:30:05.873417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls\") pod \"5f3d027f-8b99-40f7-9111-1ededbf24825\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") "
Jul 28 16:30:05.873417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert\") pod \"5f3d027f-8b99-40f7-9111-1ededbf24825\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") "
Jul 28 16:30:05.873417 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5vzf\" (UniqueName: \"kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf\") pod \"5f3d027f-8b99-40f7-9111-1ededbf24825\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") "
Jul 28 16:30:05.873715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873436    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config\") pod \"5f3d027f-8b99-40f7-9111-1ededbf24825\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") "
Jul 28 16:30:05.873715 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location\") pod \"5f3d027f-8b99-40f7-9111-1ededbf24825\" (UID: \"5f3d027f-8b99-40f7-9111-1ededbf24825\") "
Jul 28 16:30:05.873843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873789    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f3d027f-8b99-40f7-9111-1ededbf24825" (UID: "5f3d027f-8b99-40f7-9111-1ededbf24825"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:05.873843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f3d027f-8b99-40f7-9111-1ededbf24825" (UID: "5f3d027f-8b99-40f7-9111-1ededbf24825"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:05.873983 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.873869    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f3d027f-8b99-40f7-9111-1ededbf24825" (UID: "5f3d027f-8b99-40f7-9111-1ededbf24825"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:30:05.875471 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.875451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f3d027f-8b99-40f7-9111-1ededbf24825" (UID: "5f3d027f-8b99-40f7-9111-1ededbf24825"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:30:05.875597 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.875575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf" (OuterVolumeSpecName: "kube-api-access-v5vzf") pod "5f3d027f-8b99-40f7-9111-1ededbf24825" (UID: "5f3d027f-8b99-40f7-9111-1ededbf24825"). InnerVolumeSpecName "kube-api-access-v5vzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:30:05.974207 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.974173    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d027f-8b99-40f7-9111-1ededbf24825-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.974207 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.974203    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d027f-8b99-40f7-9111-1ededbf24825-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.974407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.974218    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.974407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.974230    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5vzf\" (UniqueName: \"kubernetes.io/projected/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-api-access-v5vzf\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:30:05.974407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:05.974241    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d027f-8b99-40f7-9111-1ededbf24825-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:30:06.198998 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.198964    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerID="1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e" exitCode=0
Jul 28 16:30:06.199192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.199042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"
Jul 28 16:30:06.199192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.199043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerDied","Data":"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"}
Jul 28 16:30:06.199192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.199127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5" event={"ID":"5f3d027f-8b99-40f7-9111-1ededbf24825","Type":"ContainerDied","Data":"ee4627fb9e687e4b9d44eaf525476a2ce12bac5595c31098224501393e82b626"}
Jul 28 16:30:06.199192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.199166    2579 scope.go:117] "RemoveContainer" containerID="69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"
Jul 28 16:30:06.208117 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.208098    2579 scope.go:117] "RemoveContainer" containerID="1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"
Jul 28 16:30:06.215641 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.215622    2579 scope.go:117] "RemoveContainer" containerID="328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"
Jul 28 16:30:06.221020 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.220995    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:30:06.223889 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.223858    2579 scope.go:117] "RemoveContainer" containerID="69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"
Jul 28 16:30:06.224158 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:30:06.224140    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981\": container with ID starting with 69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981 not found: ID does not exist" containerID="69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"
Jul 28 16:30:06.224222 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.224166    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981"} err="failed to get container status \"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981\": rpc error: code = NotFound desc = could not find container \"69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981\": container with ID starting with 69b4e554bd80dafd45c9fa0b52ff5d95b50f0262f75d56e648ef6c43f8315981 not found: ID does not exist"
Jul 28 16:30:06.224222 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.224181    2579 scope.go:117] "RemoveContainer" containerID="1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"
Jul 28 16:30:06.224435 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:30:06.224416    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e\": container with ID starting with 1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e not found: ID does not exist" containerID="1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"
Jul 28 16:30:06.224508 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.224454    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e"} err="failed to get container status \"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e\": rpc error: code = NotFound desc = could not find container \"1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e\": container with ID starting with 1da7eab70e490418f61020605ab015237ad5817c8a883819fa530f8db74ae28e not found: ID does not exist"
Jul 28 16:30:06.224508 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.224476    2579 scope.go:117] "RemoveContainer" containerID="328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"
Jul 28 16:30:06.224737 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:30:06.224721    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0\": container with ID starting with 328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0 not found: ID does not exist" containerID="328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"
Jul 28 16:30:06.224800 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.224742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0"} err="failed to get container status \"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0\": rpc error: code = NotFound desc = could not find container \"328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0\": container with ID starting with 328da7e483c050eec054b66caaaef6841aa9fb6d55a15e9f86ca86842b5c05e0 not found: ID does not exist"
Jul 28 16:30:06.225622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:06.225602    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7hbk5"]
Jul 28 16:30:08.015032 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:30:08.015001    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" path="/var/lib/kubelet/pods/5f3d027f-8b99-40f7-9111-1ededbf24825/volumes"
Jul 28 16:31:46.532456 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532420    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532723    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532734    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532740    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kube-rbac-proxy"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kube-rbac-proxy"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="storage-initializer"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532763    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="storage-initializer"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532817    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kserve-container"
Jul 28 16:31:46.532971 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.532832    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3d027f-8b99-40f7-9111-1ededbf24825" containerName="kube-rbac-proxy"
Jul 28 16:31:46.535780 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.535765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.538393 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:31:46.538532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538410    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:31:46.538532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5198af-kube-rbac-proxy-sar-config\""
Jul 28 16:31:46.538532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5198af-predictor-serving-cert\""
Jul 28 16:31:46.538532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:31:46.538532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:31:46.538789 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.538648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:31:46.546786 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.546763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:31:46.639730 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.639696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.639924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.639752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.639924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.639794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.639924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.639810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.639924 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.639886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndgt\" (UniqueName: \"kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741128 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741298 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndgt\" (UniqueName: \"kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741664 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.741934 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.742027 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.741998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.743585 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.743567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.750388 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.750363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndgt\" (UniqueName: \"kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt\") pod \"isvc-primary-5198af-predictor-846797f9c7-ptkcq\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") " pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.846802 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.846723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:46.980547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:46.980503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:31:46.983367 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:31:46.983341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061c79c3_9cc9_4ca6_aacb_6dc311121052.slice/crio-523f03e96dd1d67b047b5ca82fbdbf24b119a2dcdad592b757a356020a0a40c1 WatchSource:0}: Error finding container 523f03e96dd1d67b047b5ca82fbdbf24b119a2dcdad592b757a356020a0a40c1: Status 404 returned error can't find the container with id 523f03e96dd1d67b047b5ca82fbdbf24b119a2dcdad592b757a356020a0a40c1
Jul 28 16:31:47.606638 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:47.606599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerStarted","Data":"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"}
Jul 28 16:31:47.607064 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:47.606659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerStarted","Data":"523f03e96dd1d67b047b5ca82fbdbf24b119a2dcdad592b757a356020a0a40c1"}
Jul 28 16:31:51.623394 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:51.623363    2579 generic.go:356] "Generic (PLEG): container finished" podID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerID="6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55" exitCode=0
Jul 28 16:31:51.623778 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:51.623407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerDied","Data":"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"}
Jul 28 16:31:52.629868 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.629814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerStarted","Data":"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"}
Jul 28 16:31:52.630313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.629891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerStarted","Data":"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"}
Jul 28 16:31:52.630313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.630186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:52.630313 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.630244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:52.631668 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.631644    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:31:52.651227 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:52.651181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podStartSLOduration=2.8755467 podStartE2EDuration="6.651162912s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.775616212s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:52.65024139 +0000 UTC m=+1759.201839383" watchObservedRunningTime="2026-07-28 16:31:52.651162912 +0000 UTC m=+1759.202760909"
Jul 28 16:31:53.634211 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:53.634165    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:31:58.638876 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:58.638831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:31:58.639443 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:31:58.639416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:08.639756 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:08.639715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:18.640021 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:18.639982    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:28.639420 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:28.639329    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:33.965387 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:33.965357    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:32:33.967958 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:33.967937    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:32:38.639308 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:38.639266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:48.640283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:48.640235    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:32:58.639355 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:32:58.639312    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:33:08.640768 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:08.640730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:33:49.948791 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:49.948757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:33:49.951338 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:49.949014    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container" containerID="cri-o://e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264" gracePeriod=30
Jul 28 16:33:49.951338 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:49.949041    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kube-rbac-proxy" containerID="cri-o://57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd" gracePeriod=30
Jul 28 16:33:50.116359 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:50.116325    2579 generic.go:356] "Generic (PLEG): container finished" podID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerID="57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd" exitCode=2
Jul 28 16:33:50.116517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:50.116399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerDied","Data":"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"}
Jul 28 16:33:53.634866 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:53.634820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 16:33:54.698132 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.698108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:33:54.834394 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config\") pod \"061c79c3-9cc9-4ca6-aacb-6dc311121052\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") "
Jul 28 16:33:54.834394 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jndgt\" (UniqueName: \"kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt\") pod \"061c79c3-9cc9-4ca6-aacb-6dc311121052\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") "
Jul 28 16:33:54.834636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls\") pod \"061c79c3-9cc9-4ca6-aacb-6dc311121052\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") "
Jul 28 16:33:54.834636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location\") pod \"061c79c3-9cc9-4ca6-aacb-6dc311121052\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") "
Jul 28 16:33:54.834636 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert\") pod \"061c79c3-9cc9-4ca6-aacb-6dc311121052\" (UID: \"061c79c3-9cc9-4ca6-aacb-6dc311121052\") "
Jul 28 16:33:54.834784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "061c79c3-9cc9-4ca6-aacb-6dc311121052" (UID: "061c79c3-9cc9-4ca6-aacb-6dc311121052"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:54.834940 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "061c79c3-9cc9-4ca6-aacb-6dc311121052" (UID: "061c79c3-9cc9-4ca6-aacb-6dc311121052"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:33:54.835012 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834980    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:33:54.835012 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.834999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "061c79c3-9cc9-4ca6-aacb-6dc311121052" (UID: "061c79c3-9cc9-4ca6-aacb-6dc311121052"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:54.836547 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.836527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "061c79c3-9cc9-4ca6-aacb-6dc311121052" (UID: "061c79c3-9cc9-4ca6-aacb-6dc311121052"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:33:54.836635 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.836596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt" (OuterVolumeSpecName: "kube-api-access-jndgt") pod "061c79c3-9cc9-4ca6-aacb-6dc311121052" (UID: "061c79c3-9cc9-4ca6-aacb-6dc311121052"). InnerVolumeSpecName "kube-api-access-jndgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:33:54.936391 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.936355    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndgt\" (UniqueName: \"kubernetes.io/projected/061c79c3-9cc9-4ca6-aacb-6dc311121052-kube-api-access-jndgt\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:33:54.936391 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.936386    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/061c79c3-9cc9-4ca6-aacb-6dc311121052-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:33:54.936391 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.936398    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/061c79c3-9cc9-4ca6-aacb-6dc311121052-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:33:54.936629 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:54.936406    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/061c79c3-9cc9-4ca6-aacb-6dc311121052-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:33:55.138564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.138467    2579 generic.go:356] "Generic (PLEG): container finished" podID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerID="e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264" exitCode=0
Jul 28 16:33:55.138564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.138518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerDied","Data":"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"}
Jul 28 16:33:55.138564 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.138530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"
Jul 28 16:33:55.138863 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.138582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq" event={"ID":"061c79c3-9cc9-4ca6-aacb-6dc311121052","Type":"ContainerDied","Data":"523f03e96dd1d67b047b5ca82fbdbf24b119a2dcdad592b757a356020a0a40c1"}
Jul 28 16:33:55.138863 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.138618    2579 scope.go:117] "RemoveContainer" containerID="57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"
Jul 28 16:33:55.147164 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.147143    2579 scope.go:117] "RemoveContainer" containerID="e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"
Jul 28 16:33:55.154655 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.154636    2579 scope.go:117] "RemoveContainer" containerID="6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"
Jul 28 16:33:55.162236 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.162216    2579 scope.go:117] "RemoveContainer" containerID="57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"
Jul 28 16:33:55.162740 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:33:55.162507    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd\": container with ID starting with 57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd not found: ID does not exist" containerID="57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"
Jul 28 16:33:55.162740 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.162553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd"} err="failed to get container status \"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd\": rpc error: code = NotFound desc = could not find container \"57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd\": container with ID starting with 57e10e76c7b3beeb3812e38a4f733ec6d73b7c722473efac1b5bb72c8b0931cd not found: ID does not exist"
Jul 28 16:33:55.162740 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.162577    2579 scope.go:117] "RemoveContainer" containerID="e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"
Jul 28 16:33:55.163009 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:33:55.162989    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264\": container with ID starting with e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264 not found: ID does not exist" containerID="e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"
Jul 28 16:33:55.163067 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.163020    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264"} err="failed to get container status \"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264\": rpc error: code = NotFound desc = could not find container \"e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264\": container with ID starting with e1944cc63e5246acf8d45a0ca7709c0c68726085de3aca748e9b4d08fbc78264 not found: ID does not exist"
Jul 28 16:33:55.163067 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.163044    2579 scope.go:117] "RemoveContainer" containerID="6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"
Jul 28 16:33:55.163325 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:33:55.163295    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55\": container with ID starting with 6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55 not found: ID does not exist" containerID="6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"
Jul 28 16:33:55.163398 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.163323    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55"} err="failed to get container status \"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55\": rpc error: code = NotFound desc = could not find container \"6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55\": container with ID starting with 6e64f25eb4aed537e9a17d7c741cdfe17bd4e2e5f4221acf8f5322d9d3faae55 not found: ID does not exist"
Jul 28 16:33:55.164406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.164388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:33:55.169012 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:55.168992    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5198af-predictor-846797f9c7-ptkcq"]
Jul 28 16:33:56.014829 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:33:56.014788    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" path="/var/lib/kubelet/pods/061c79c3-9cc9-4ca6-aacb-6dc311121052/volumes"
Jul 28 16:35:40.676385 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676674    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kube-rbac-proxy"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676679    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kube-rbac-proxy"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676694    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="storage-initializer"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676699    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="storage-initializer"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676749    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kserve-container"
Jul 28 16:35:40.676797 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.676756    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="061c79c3-9cc9-4ca6-aacb-6dc311121052" containerName="kube-rbac-proxy"
Jul 28 16:35:40.679600 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.679582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.681960 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.681930    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-384b7-kube-rbac-proxy-sar-config\""
Jul 28 16:35:40.682077 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.681937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:35:40.682338 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.682320    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:35:40.682407 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.682371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-384b7-predictor-serving-cert\""
Jul 28 16:35:40.682462 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.682379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:35:40.682517 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.682492    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:35:40.682684 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.682667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-m9t97\""
Jul 28 16:35:40.692182 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.692160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:35:40.837621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.837568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.837621 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.837633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.837869 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.837695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.837869 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.837741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxbs\" (UniqueName: \"kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.837869 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.837770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.938904 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.938871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.938922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.938940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.939056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxbs\" (UniqueName: \"kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939172 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.939096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939359 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.939328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939722 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.939699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.939781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.939732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.941432 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.941413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.948570 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.948550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxbs\" (UniqueName: \"kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs\") pod \"raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:40.990294 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:40.990266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:41.124696 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:41.124523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:35:41.127369 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:35:41.127341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7eca32_f2e7_4235_b294_21fe8a8c9d4c.slice/crio-534acf7f4c87f0ba51c103a2d910fe564d079435d15faab15b402b553baab34b WatchSource:0}: Error finding container 534acf7f4c87f0ba51c103a2d910fe564d079435d15faab15b402b553baab34b: Status 404 returned error can't find the container with id 534acf7f4c87f0ba51c103a2d910fe564d079435d15faab15b402b553baab34b
Jul 28 16:35:41.570053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:41.570021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerStarted","Data":"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"}
Jul 28 16:35:41.570217 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:41.570075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerStarted","Data":"534acf7f4c87f0ba51c103a2d910fe564d079435d15faab15b402b553baab34b"}
Jul 28 16:35:45.587059 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:45.587025    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerID="ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0" exitCode=0
Jul 28 16:35:45.587433 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:45.587100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerDied","Data":"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"}
Jul 28 16:35:46.593065 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.593032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerStarted","Data":"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"}
Jul 28 16:35:46.593438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.593093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerStarted","Data":"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"}
Jul 28 16:35:46.593438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.593398    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:46.593548 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.593531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:46.594893 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.594867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:35:46.613986 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:46.613941    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podStartSLOduration=2.919536 podStartE2EDuration="6.613913405s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.694377405s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:35:46.611808342 +0000 UTC m=+1993.163406344" watchObservedRunningTime="2026-07-28 16:35:46.613913405 +0000 UTC m=+1993.165511398"
Jul 28 16:35:47.597081 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:47.597041    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:35:52.601660 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:52.601632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:35:52.602182 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:35:52.602155    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:02.602120 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:02.602081    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:12.603085 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:12.603045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:22.602646 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:22.602592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:32.602260 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:32.602224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:42.602668 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:42.602623    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:52.602896 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:52.602834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:36:56.009692 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:36:56.009594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:37:06.014539 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:06.014506    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:37:10.867450 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:10.867415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:37:10.867858 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:10.867693    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container" containerID="cri-o://490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991" gracePeriod=30
Jul 28 16:37:10.867858 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:10.867774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kube-rbac-proxy" containerID="cri-o://ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149" gracePeriod=30
Jul 28 16:37:11.945767 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:11.945733    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerID="ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149" exitCode=2
Jul 28 16:37:11.946198 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:11.945804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerDied","Data":"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"}
Jul 28 16:37:12.597231 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:12.597192    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 16:37:15.612537 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.612511    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:37:15.762199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location\") pod \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") "
Jul 28 16:37:15.762199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert\") pod \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") "
Jul 28 16:37:15.762199 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762171    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls\") pod \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") "
Jul 28 16:37:15.762494 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtxbs\" (UniqueName: \"kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs\") pod \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") "
Jul 28 16:37:15.762494 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config\") pod \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\" (UID: \"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c\") "
Jul 28 16:37:15.762605 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" (UID: "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:37:15.762659 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" (UID: "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:37:15.762697 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.762657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" (UID: "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:37:15.764366 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.764343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" (UID: "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:37:15.764503 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.764475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs" (OuterVolumeSpecName: "kube-api-access-jtxbs") pod "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" (UID: "dd7eca32-f2e7-4235-b294-21fe8a8c9d4c"). InnerVolumeSpecName "kube-api-access-jtxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:37:15.863269 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.863231    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kserve-provision-location\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:15.863269 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.863263    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-cabundle-cert\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:15.863269 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.863276    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-proxy-tls\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:15.863514 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.863288    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtxbs\" (UniqueName: \"kubernetes.io/projected/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-api-access-jtxbs\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:15.863514 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.863299    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:15.964352 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.964318    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerID="490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991" exitCode=0
Jul 28 16:37:15.964567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.964384    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"
Jul 28 16:37:15.964567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.964401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerDied","Data":"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"}
Jul 28 16:37:15.964567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.964490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz" event={"ID":"dd7eca32-f2e7-4235-b294-21fe8a8c9d4c","Type":"ContainerDied","Data":"534acf7f4c87f0ba51c103a2d910fe564d079435d15faab15b402b553baab34b"}
Jul 28 16:37:15.964567 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.964518    2579 scope.go:117] "RemoveContainer" containerID="ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"
Jul 28 16:37:15.973529 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.973512    2579 scope.go:117] "RemoveContainer" containerID="490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"
Jul 28 16:37:15.980683 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.980666    2579 scope.go:117] "RemoveContainer" containerID="ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"
Jul 28 16:37:15.987515 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.987488    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:37:15.987720 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.987703    2579 scope.go:117] "RemoveContainer" containerID="ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"
Jul 28 16:37:15.988033 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:37:15.988015    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149\": container with ID starting with ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149 not found: ID does not exist" containerID="ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"
Jul 28 16:37:15.988089 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.988043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149"} err="failed to get container status \"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149\": rpc error: code = NotFound desc = could not find container \"ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149\": container with ID starting with ce20513a3a5b7f89a3215cf1ffaa060b1a88a3e32534459591872bfb58513149 not found: ID does not exist"
Jul 28 16:37:15.988089 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.988061    2579 scope.go:117] "RemoveContainer" containerID="490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"
Jul 28 16:37:15.988311 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:37:15.988294    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991\": container with ID starting with 490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991 not found: ID does not exist" containerID="490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"
Jul 28 16:37:15.988352 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.988316    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991"} err="failed to get container status \"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991\": rpc error: code = NotFound desc = could not find container \"490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991\": container with ID starting with 490de4f36ad7cb1f08c9393f5a8a2de6cc709058ac862aefb2e7e88430a5f991 not found: ID does not exist"
Jul 28 16:37:15.988352 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.988331    2579 scope.go:117] "RemoveContainer" containerID="ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"
Jul 28 16:37:15.988558 ip-10-0-142-59 kubenswrapper[2579]: E0728 16:37:15.988539    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0\": container with ID starting with ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0 not found: ID does not exist" containerID="ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"
Jul 28 16:37:15.988618 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.988566    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0"} err="failed to get container status \"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0\": rpc error: code = NotFound desc = could not find container \"ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0\": container with ID starting with ebf300005713d497193599e9e0d0c364b6c05f47a2a781aabfd312aacb47d6e0 not found: ID does not exist"
Jul 28 16:37:15.995380 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:15.995359    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-384b7-predictor-7b9b688c5b-q5vvz"]
Jul 28 16:37:16.014671 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.014610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" path="/var/lib/kubelet/pods/dd7eca32-f2e7-4235-b294-21fe8a8c9d4c/volumes"
Jul 28 16:37:16.864359 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zchs2/must-gather-mt9t9"]
Jul 28 16:37:16.864932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864911    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kube-rbac-proxy"
Jul 28 16:37:16.864932 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kube-rbac-proxy"
Jul 28 16:37:16.865082 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864968    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container"
Jul 28 16:37:16.865082 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864977    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container"
Jul 28 16:37:16.865082 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.864993    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="storage-initializer"
Jul 28 16:37:16.865082 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.865001    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="storage-initializer"
Jul 28 16:37:16.865082 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.865074    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kserve-container"
Jul 28 16:37:16.865332 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.865089    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7eca32-f2e7-4235-b294-21fe8a8c9d4c" containerName="kube-rbac-proxy"
Jul 28 16:37:16.869293 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.869272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:16.871481 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.871462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zchs2\"/\"kube-root-ca.crt\""
Jul 28 16:37:16.871965 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.871947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zchs2\"/\"openshift-service-ca.crt\""
Jul 28 16:37:16.872041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.871949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zchs2\"/\"default-dockercfg-4hltm\""
Jul 28 16:37:16.883283 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.883257    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zchs2/must-gather-mt9t9"]
Jul 28 16:37:16.972718 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.972684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h72w\" (UniqueName: \"kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:16.972910 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:16.972740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.073594 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.073555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.073782 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.073651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h72w\" (UniqueName: \"kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.073905 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.073885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.084383 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.084356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h72w\" (UniqueName: \"kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w\") pod \"must-gather-mt9t9\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") " pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.197428 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.197394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:17.337745 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.337703    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zchs2/must-gather-mt9t9"]
Jul 28 16:37:17.341586 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:37:17.341561    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8335177d_b2cd_45b7_9952_2e3e733e4e93.slice/crio-bc3d3acc4340ada9d309b9e06122c6199032e8fd1b3161d023bc957f68e6a908 WatchSource:0}: Error finding container bc3d3acc4340ada9d309b9e06122c6199032e8fd1b3161d023bc957f68e6a908: Status 404 returned error can't find the container with id bc3d3acc4340ada9d309b9e06122c6199032e8fd1b3161d023bc957f68e6a908
Jul 28 16:37:17.343780 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.343764    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:37:17.977763 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:17.977697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zchs2/must-gather-mt9t9" event={"ID":"8335177d-b2cd-45b7-9952-2e3e733e4e93","Type":"ContainerStarted","Data":"bc3d3acc4340ada9d309b9e06122c6199032e8fd1b3161d023bc957f68e6a908"}
Jul 28 16:37:24.018660 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:24.018608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zchs2/must-gather-mt9t9" event={"ID":"8335177d-b2cd-45b7-9952-2e3e733e4e93","Type":"ContainerStarted","Data":"faab18b961bc04c0446edacae3a47aec87e4b2e5f7d9af5bac1f6bfbfaf954ed"}
Jul 28 16:37:24.019125 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:24.018669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zchs2/must-gather-mt9t9" event={"ID":"8335177d-b2cd-45b7-9952-2e3e733e4e93","Type":"ContainerStarted","Data":"21c0e6c312342c93e4f127832af180247ff40682486f020c2885d09a66946204"}
Jul 28 16:37:24.041745 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:24.041695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zchs2/must-gather-mt9t9" podStartSLOduration=2.035937796 podStartE2EDuration="8.041681002s" totalImagesPullingTime="6.005743206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:37:24.040211637 +0000 UTC m=+2090.591809631" watchObservedRunningTime="2026-07-28 16:37:24.041681002 +0000 UTC m=+2090.593278995"
Jul 28 16:37:33.995753 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:33.995718    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:37:33.996259 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:33.996155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:37:46.122474 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:46.122444    2579 generic.go:356] "Generic (PLEG): container finished" podID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerID="21c0e6c312342c93e4f127832af180247ff40682486f020c2885d09a66946204" exitCode=0
Jul 28 16:37:46.123037 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:46.122527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zchs2/must-gather-mt9t9" event={"ID":"8335177d-b2cd-45b7-9952-2e3e733e4e93","Type":"ContainerDied","Data":"21c0e6c312342c93e4f127832af180247ff40682486f020c2885d09a66946204"}
Jul 28 16:37:46.123037 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:46.122976    2579 scope.go:117] "RemoveContainer" containerID="21c0e6c312342c93e4f127832af180247ff40682486f020c2885d09a66946204"
Jul 28 16:37:46.507168 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:46.507145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zchs2_must-gather-mt9t9_8335177d-b2cd-45b7-9952-2e3e733e4e93/gather/0.log"
Jul 28 16:37:47.269246 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.269212    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7d85/must-gather-mgpzd"]
Jul 28 16:37:47.271648 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.271632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.275366 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.275344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7d85\"/\"kube-root-ca.crt\""
Jul 28 16:37:47.275481 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.275465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7d85\"/\"openshift-service-ca.crt\""
Jul 28 16:37:47.288177 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.288153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/must-gather-mgpzd"]
Jul 28 16:37:47.360186 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.360158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57dbbf27-f7f4-45f8-8554-520842c187b1-must-gather-output\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.360354 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.360246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h74q\" (UniqueName: \"kubernetes.io/projected/57dbbf27-f7f4-45f8-8554-520842c187b1-kube-api-access-5h74q\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.461434 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.461398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57dbbf27-f7f4-45f8-8554-520842c187b1-must-gather-output\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.461627 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.461464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h74q\" (UniqueName: \"kubernetes.io/projected/57dbbf27-f7f4-45f8-8554-520842c187b1-kube-api-access-5h74q\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.461837 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.461812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57dbbf27-f7f4-45f8-8554-520842c187b1-must-gather-output\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.471385 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.471358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h74q\" (UniqueName: \"kubernetes.io/projected/57dbbf27-f7f4-45f8-8554-520842c187b1-kube-api-access-5h74q\") pod \"must-gather-mgpzd\" (UID: \"57dbbf27-f7f4-45f8-8554-520842c187b1\") " pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.581073 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.580973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/must-gather-mgpzd"
Jul 28 16:37:47.711556 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:47.711524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/must-gather-mgpzd"]
Jul 28 16:37:47.714477 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:37:47.714448    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57dbbf27_f7f4_45f8_8554_520842c187b1.slice/crio-813c88ab39a3c9ca8e2cbb3f8474895b31a7ddc6e199b6d6e71b660adaf0a7ab WatchSource:0}: Error finding container 813c88ab39a3c9ca8e2cbb3f8474895b31a7ddc6e199b6d6e71b660adaf0a7ab: Status 404 returned error can't find the container with id 813c88ab39a3c9ca8e2cbb3f8474895b31a7ddc6e199b6d6e71b660adaf0a7ab
Jul 28 16:37:48.131216 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:48.131176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/must-gather-mgpzd" event={"ID":"57dbbf27-f7f4-45f8-8554-520842c187b1","Type":"ContainerStarted","Data":"813c88ab39a3c9ca8e2cbb3f8474895b31a7ddc6e199b6d6e71b660adaf0a7ab"}
Jul 28 16:37:49.138836 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:49.138804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/must-gather-mgpzd" event={"ID":"57dbbf27-f7f4-45f8-8554-520842c187b1","Type":"ContainerStarted","Data":"47f856fe77a78d471dbbf0c775cc8171598bf94c0f7498347621714308ad1506"}
Jul 28 16:37:49.139319 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:49.138843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/must-gather-mgpzd" event={"ID":"57dbbf27-f7f4-45f8-8554-520842c187b1","Type":"ContainerStarted","Data":"1e683db2296d6763fb0762d52f2963a11d491797dc480dd1a3da5bdf708aacdb"}
Jul 28 16:37:49.165714 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:49.165645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7d85/must-gather-mgpzd" podStartSLOduration=1.30986549 podStartE2EDuration="2.165620071s" totalImagesPullingTime="855.754581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:37:49.164023842 +0000 UTC m=+2115.715621836" watchObservedRunningTime="2026-07-28 16:37:49.165620071 +0000 UTC m=+2115.717218262"
Jul 28 16:37:50.139468 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:50.139434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jmktt_8109b844-aca8-4f61-b52c-61dc7132a484/global-pull-secret-syncer/0.log"
Jul 28 16:37:50.406099 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:50.406022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xqxd6_292976e5-5217-4dc6-8804-ea7015cb146d/konnectivity-agent/0.log"
Jul 28 16:37:50.512553 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:50.512525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-59.ec2.internal_d42031d4bd857d1b7419a26ede69f8dc/haproxy/0.log"
Jul 28 16:37:51.957209 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:51.957171    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zchs2/must-gather-mt9t9"]
Jul 28 16:37:51.957678 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:51.957455    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zchs2/must-gather-mt9t9" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="copy" containerID="cri-o://faab18b961bc04c0446edacae3a47aec87e4b2e5f7d9af5bac1f6bfbfaf954ed" gracePeriod=2
Jul 28 16:37:51.991356 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:51.991316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zchs2/must-gather-mt9t9"]
Jul 28 16:37:52.157053 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.156968    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zchs2_must-gather-mt9t9_8335177d-b2cd-45b7-9952-2e3e733e4e93/copy/0.log"
Jul 28 16:37:52.157661 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.157598    2579 generic.go:356] "Generic (PLEG): container finished" podID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerID="faab18b961bc04c0446edacae3a47aec87e4b2e5f7d9af5bac1f6bfbfaf954ed" exitCode=143
Jul 28 16:37:52.378044 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.378015    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zchs2_must-gather-mt9t9_8335177d-b2cd-45b7-9952-2e3e733e4e93/copy/0.log"
Jul 28 16:37:52.380954 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.380927    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:52.517406 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.517364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h72w\" (UniqueName: \"kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w\") pod \"8335177d-b2cd-45b7-9952-2e3e733e4e93\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") "
Jul 28 16:37:52.517685 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.517501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output\") pod \"8335177d-b2cd-45b7-9952-2e3e733e4e93\" (UID: \"8335177d-b2cd-45b7-9952-2e3e733e4e93\") "
Jul 28 16:37:52.519075 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.519037    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8335177d-b2cd-45b7-9952-2e3e733e4e93" (UID: "8335177d-b2cd-45b7-9952-2e3e733e4e93"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:37:52.527179 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.523281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w" (OuterVolumeSpecName: "kube-api-access-4h72w") pod "8335177d-b2cd-45b7-9952-2e3e733e4e93" (UID: "8335177d-b2cd-45b7-9952-2e3e733e4e93"). InnerVolumeSpecName "kube-api-access-4h72w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:37:52.618843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.618739    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h72w\" (UniqueName: \"kubernetes.io/projected/8335177d-b2cd-45b7-9952-2e3e733e4e93-kube-api-access-4h72w\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:52.618843 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:52.618792    2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8335177d-b2cd-45b7-9952-2e3e733e4e93-must-gather-output\") on node \"ip-10-0-142-59.ec2.internal\" DevicePath \"\""
Jul 28 16:37:53.165875 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:53.165823    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zchs2_must-gather-mt9t9_8335177d-b2cd-45b7-9952-2e3e733e4e93/copy/0.log"
Jul 28 16:37:53.166448 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:53.166423    2579 scope.go:117] "RemoveContainer" containerID="faab18b961bc04c0446edacae3a47aec87e4b2e5f7d9af5bac1f6bfbfaf954ed"
Jul 28 16:37:53.166622 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:53.166605    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zchs2/must-gather-mt9t9"
Jul 28 16:37:53.180168 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:53.180141    2579 scope.go:117] "RemoveContainer" containerID="21c0e6c312342c93e4f127832af180247ff40682486f020c2885d09a66946204"
Jul 28 16:37:54.023048 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:54.022981    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" path="/var/lib/kubelet/pods/8335177d-b2cd-45b7-9952-2e3e733e4e93/volumes"
Jul 28 16:37:55.276705 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.276668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-j7wxw_a304a7f1-175e-45e2-8d83-d2b7a791c434/cluster-monitoring-operator/0.log"
Jul 28 16:37:55.447294 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.447270    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c86664fdb-5pq5p_7e5a56f7-335f-4258-8505-d259e2b34c5f/metrics-server/0.log"
Jul 28 16:37:55.761213 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.761163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhtwm_54375cb1-27ea-4ab7-b107-68a425ab4db0/node-exporter/0.log"
Jul 28 16:37:55.796874 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.796811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhtwm_54375cb1-27ea-4ab7-b107-68a425ab4db0/kube-rbac-proxy/0.log"
Jul 28 16:37:55.832680 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.832598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhtwm_54375cb1-27ea-4ab7-b107-68a425ab4db0/init-textfile/0.log"
Jul 28 16:37:55.884750 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.884717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hd9xj_2c88adca-ca36-4788-8369-161058283316/kube-rbac-proxy-main/0.log"
Jul 28 16:37:55.933532 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.933472    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hd9xj_2c88adca-ca36-4788-8369-161058283316/kube-rbac-proxy-self/0.log"
Jul 28 16:37:55.988781 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:55.988660    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hd9xj_2c88adca-ca36-4788-8369-161058283316/openshift-state-metrics/0.log"
Jul 28 16:37:56.074290 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.074262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/prometheus/0.log"
Jul 28 16:37:56.129703 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.129675    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/config-reloader/0.log"
Jul 28 16:37:56.192293 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.192270    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/thanos-sidecar/0.log"
Jul 28 16:37:56.228285 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.228260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/kube-rbac-proxy-web/0.log"
Jul 28 16:37:56.262707 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.262619    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/kube-rbac-proxy/0.log"
Jul 28 16:37:56.301604 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.301571    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/kube-rbac-proxy-thanos/0.log"
Jul 28 16:37:56.342910 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.342875    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_185512b9-c6ad-499f-828e-50bc27899e4c/init-config-reloader/0.log"
Jul 28 16:37:56.395923 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.395889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-jmvqq_58014279-5258-496b-afd5-c6eda3e84e50/prometheus-operator/0.log"
Jul 28 16:37:56.438658 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.438607    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-jmvqq_58014279-5258-496b-afd5-c6eda3e84e50/kube-rbac-proxy/0.log"
Jul 28 16:37:56.540830 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.540750    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56599ff8b9-68b27_0999f069-f015-4c7a-b7e4-96cd466835ee/telemeter-client/0.log"
Jul 28 16:37:56.583482 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.583432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56599ff8b9-68b27_0999f069-f015-4c7a-b7e4-96cd466835ee/reload/0.log"
Jul 28 16:37:56.620142 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:56.620110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56599ff8b9-68b27_0999f069-f015-4c7a-b7e4-96cd466835ee/kube-rbac-proxy/0.log"
Jul 28 16:37:57.864774 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.864747    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"]
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865111    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="copy"
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865122    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="copy"
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865133    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="gather"
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="gather"
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865193    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="copy"
Jul 28 16:37:57.865277 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.865203    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8335177d-b2cd-45b7-9952-2e3e733e4e93" containerName="gather"
Jul 28 16:37:57.868463 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.868384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:57.870724 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.870697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7d85\"/\"default-dockercfg-zs968\""
Jul 28 16:37:57.882666 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.882633    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"]
Jul 28 16:37:57.970805 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.970774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-sys\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:57.971014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.970827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-lib-modules\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:57.971014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.970865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-podres\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:57.971014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.970892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtf7\" (UniqueName: \"kubernetes.io/projected/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-kube-api-access-bgtf7\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:57.971014 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:57.970926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-proc\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.013530 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.013502    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-tznhk_4ed9dca5-b538-4613-a9d3-6da6a3a1e731/networking-console-plugin/0.log"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-sys\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-lib-modules\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-podres\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtf7\" (UniqueName: \"kubernetes.io/projected/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-kube-api-access-bgtf7\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-proc\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-proc\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-sys\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-lib-modules\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.072225 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.071991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-podres\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.085396 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.085362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtf7\" (UniqueName: \"kubernetes.io/projected/c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c-kube-api-access-bgtf7\") pod \"perf-node-gather-daemonset-vh7ll\" (UID: \"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c\") " pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.180484 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.180460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:58.347041 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.346994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"]
Jul 28 16:37:58.352887 ip-10-0-142-59 kubenswrapper[2579]: W0728 16:37:58.351995    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0c3a952_82fe_4d7e_8aa3_f6c0cecf708c.slice/crio-548ddd922b27e7b27a9ae11ba1a213a89f4adc16278d6d59ef5406bc462e09b7 WatchSource:0}: Error finding container 548ddd922b27e7b27a9ae11ba1a213a89f4adc16278d6d59ef5406bc462e09b7: Status 404 returned error can't find the container with id 548ddd922b27e7b27a9ae11ba1a213a89f4adc16278d6d59ef5406bc462e09b7
Jul 28 16:37:58.633529 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.633477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/1.log"
Jul 28 16:37:58.638670 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:58.638637    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vbvkl_139a823e-a65d-4fd2-add0-ee066edd494b/console-operator/2.log"
Jul 28 16:37:59.120961 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.120938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77c6cf54b8-tbn6f_3aad0404-b197-4263-8438-f419bed36de6/console/0.log"
Jul 28 16:37:59.168761 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.168734    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-8btsp_405fca12-c251-41b9-92d7-41e987525982/download-server/0.log"
Jul 28 16:37:59.202802 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.202760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll" event={"ID":"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c","Type":"ContainerStarted","Data":"0ad44f14f41029dc92513127235a46fad69c3a393f53e8ac49b4a3e52e7efdbb"}
Jul 28 16:37:59.203003 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.202818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll" event={"ID":"c0c3a952-82fe-4d7e-8aa3-f6c0cecf708c","Type":"ContainerStarted","Data":"548ddd922b27e7b27a9ae11ba1a213a89f4adc16278d6d59ef5406bc462e09b7"}
Jul 28 16:37:59.203003 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.202900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:37:59.242403 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.242344    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll" podStartSLOduration=2.242320846 podStartE2EDuration="2.242320846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:37:59.240291286 +0000 UTC m=+2125.791889282" watchObservedRunningTime="2026-07-28 16:37:59.242320846 +0000 UTC m=+2125.793918841"
Jul 28 16:37:59.846054 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:37:59.846024    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-4t4j8_75e6180b-9257-4a8d-8b42-30aadcc3c76a/volume-data-source-validator/0.log"
Jul 28 16:38:00.937616 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:00.937588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5srrc_cb4d19b7-6b21-462a-9db2-11febdef81e4/dns/0.log"
Jul 28 16:38:00.985438 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:00.985408    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5srrc_cb4d19b7-6b21-462a-9db2-11febdef81e4/kube-rbac-proxy/0.log"
Jul 28 16:38:01.058470 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:01.058430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqjnl_03033695-3e53-43bc-b5e7-c121d4dd9fe0/dns-node-resolver/0.log"
Jul 28 16:38:01.665013 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:01.664988    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c6pdm_86b6b215-9318-4e1d-b112-6fb7eb1e5025/node-ca/0.log"
Jul 28 16:38:03.026525 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:03.026498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ph7dx_d57eebac-dec7-4853-b9e0-420e03e1bd5c/serve-healthcheck-canary/0.log"
Jul 28 16:38:03.579191 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:03.579162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfgp6_f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec/kube-rbac-proxy/0.log"
Jul 28 16:38:03.610911 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:03.610885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfgp6_f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec/exporter/0.log"
Jul 28 16:38:03.652999 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:03.652974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfgp6_f691cc0e-5d31-44ad-a29a-ea1f5fe2c3ec/extractor/0.log"
Jul 28 16:38:05.217914 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:05.217888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j7d85/perf-node-gather-daemonset-vh7ll"
Jul 28 16:38:06.383373 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:06.383348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tx9nx_f624a898-05de-4aa7-993a-8790c6c035b3/s3-init/0.log"
Jul 28 16:38:06.403426 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:06.403401    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tx9nx_f624a898-05de-4aa7-993a-8790c6c035b3/download-hf-model/0.log"
Jul 28 16:38:11.760784 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:11.760757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8_f3be8fcf-eef5-4e7e-b55e-256eae8b17b6/kube-storage-version-migrator-operator/0.log"
Jul 28 16:38:11.761192 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:11.760960    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vlzl8_f3be8fcf-eef5-4e7e-b55e-256eae8b17b6/kube-storage-version-migrator-operator/1.log"
Jul 28 16:38:12.777906 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:12.777867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4zhn2_201d3cf9-6472-422b-b45f-a833061e3916/kube-multus/0.log"
Jul 28 16:38:13.311996 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.311966    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:38:13.343917 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.343890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/egress-router-binary-copy/0.log"
Jul 28 16:38:13.374048 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.374019    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/cni-plugins/0.log"
Jul 28 16:38:13.402629 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.402600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/bond-cni-plugin/0.log"
Jul 28 16:38:13.443166 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.443140    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/routeoverride-cni/0.log"
Jul 28 16:38:13.489205 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.489171    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/whereabouts-cni-bincopy/0.log"
Jul 28 16:38:13.533301 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.533268    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvgkp_4b0c4aa6-0a2d-4fa6-9746-9ac4c315af4f/whereabouts-cni/0.log"
Jul 28 16:38:13.706280 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.706253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ztqjn_0e77f42d-6f86-4e08-b27e-ce9f7323662b/network-metrics-daemon/0.log"
Jul 28 16:38:13.739958 ip-10-0-142-59 kubenswrapper[2579]: I0728 16:38:13.739930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ztqjn_0e77f42d-6f86-4e08-b27e-ce9f7323662b/kube-rbac-proxy/0.log"