Jul 01 18:35:26.353459 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 18:35:26.353469 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 18:35:26.353477 ip-10-0-137-241 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 18:35:26.353698 ip-10-0-137-241 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 18:35:36.602436 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 18:35:36.602458 ip-10-0-137-241 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b6405f4b107743c69ab307721c754018 --
Jul 01 18:37:20.579086 ip-10-0-137-241 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.912555    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919413    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919427    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919431    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919434    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919437    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919440    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919444    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:20.978736 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919447    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919450    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919453    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919456    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919459    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919462    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919465    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919468    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919470    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919473    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919478    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919482    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919484    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919488    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919492    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919495    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919499    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919503    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.203686 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919506    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:20.980995 ip-10-0-137-241 systemd[1]: Started Kubernetes Kubelet.
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919509    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919512    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919515    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919518    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919521    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919526    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919529    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919531    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919534    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919536    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919539    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919541    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919544    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919546    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919549    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919552    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919555    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919557    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919560    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.212076 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919562    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919565    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919567    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919570    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919572    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919575    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919577    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919581    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919583    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919586    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919588    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919591    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919594    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919596    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919599    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919602    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919604    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919608    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919612    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.213791 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919614    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919617    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919619    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919622    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919624    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919627    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919629    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919632    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919634    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919636    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919640    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919642    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919645    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919648    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919654    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919656    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919659    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919665    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919668    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919671    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.215845 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919673    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919676    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919678    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919680    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919683    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919686    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919688    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919691    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919693    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919696    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919699    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919701    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919703    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919706    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.919708    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920387    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920393    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920396    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920399    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920402    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.216895 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920404    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920407    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920412    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920416    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920419    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920421    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920424    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920427    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920430    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920435    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920438    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920441    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920445    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920448    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920451    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920453    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920456    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920459    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920461    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920464    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.218763 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920467    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920470    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920472    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920475    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920478    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920480    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920483    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920486    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920489    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920491    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920494    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920496    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920499    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920502    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920504    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920507    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920509    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920512    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920515    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920519    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.220204 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920522    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920527    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920530    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920533    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920536    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920538    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920541    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920543    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920547    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920549    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920552    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920554    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920557    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920559    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920562    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920564    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920567    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920569    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920572    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920575    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.221336 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920578    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920580    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920583    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920585    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920588    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920591    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920594    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920597    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920600    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920603    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920605    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920608    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920610    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920614    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920616    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920620    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920623    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920626    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920629    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.222146 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920631    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920634    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920636    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920639    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920641    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920644    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920648    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920650    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920652    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920655    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920658    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920660    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920663    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920666    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.920668    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921046    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921052    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921056    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921059    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.222752 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921062    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921065    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921069    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921072    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921076    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921079    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921081    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921086    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921089    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921093    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921095    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921098    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921101    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921104    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921106    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921109    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921111    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921114    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921116    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.223386 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921119    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921122    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921124    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921127    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921129    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921131    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921134    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921136    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921139    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921141    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921144    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921146    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921149    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921152    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921155    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921157    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921160    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921162    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921164    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921167    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:21.223992 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921171    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921173    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921177    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921179    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921182    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921184    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921187    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921190    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921192    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921197    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921199    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921202    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921204    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921207    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921209    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921211    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921215    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921218    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921221    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921224    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.224601 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921226    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921229    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921231    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921234    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921236    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921239    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921241    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921243    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921246    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921248    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921251    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921253    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921257    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921259    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921263    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921265    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921268    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921270    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921273    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921276    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.225435 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921278    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921281    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921283    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921286    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921288    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921291    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921293    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921295    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921298    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921300    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921303    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921305    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921309    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921312    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921314    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921317    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921379    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921385    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921392    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921396    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921400    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 18:37:21.226012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921403    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921406    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921410    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921415    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921417    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921422    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921425    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921428    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921431    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921434    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921437    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921440    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921443    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921447    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921450    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921453    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921456    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921459    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921462    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921465    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921468    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921471    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921474    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921477    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 18:37:21.226686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921480    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921483    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921486    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921489    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921492    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921495    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921498    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921501    2583 flags.go:64] FLAG: --enable-server="true"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921503    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921507    2583 flags.go:64] FLAG: --event-burst="100"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921511    2583 flags.go:64] FLAG: --event-qps="50"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921514    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921517    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921521    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921525    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921527    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921530    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921533    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921536    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921539    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921542    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921545    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921548    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921551    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921553    2583 flags.go:64] FLAG: --feature-gates=""
Jul 01 18:37:21.227301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921557    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921560    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921563    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921566    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921569    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921571    2583 flags.go:64] FLAG: --help="false"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921574    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921577    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921580    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921582    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921585    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921589    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921592    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921594    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921597    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921599    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921602    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921605    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921609    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921612    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921615    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921618    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921621    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921624    2583 flags.go:64] FLAG: --lock-file=""
Jul 01 18:37:21.228054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921626    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921629    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921632    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921638    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921641    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921644    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921647    2583 flags.go:64] FLAG: --logging-format="text"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921649    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921653    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921655    2583 flags.go:64] FLAG: --manifest-url=""
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921658    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921662    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921665    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921669    2583 flags.go:64] FLAG: --max-pods="110"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921671    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921674    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921677    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921679    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921682    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921685    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921688    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921694    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921697    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921700    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 18:37:21.228681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921703    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921705    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921710    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921714    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921718    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921721    2583 flags.go:64] FLAG: --port="10250"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921724    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921727    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a85d42ee5fb080d3"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921730    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921733    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921735    2583 flags.go:64] FLAG: --register-node="true"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921738    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921741    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921745    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921748    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921751    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921756    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921759    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921762    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921765    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921768    2583 flags.go:64] FLAG: --runonce="false"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921771    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921773    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921776    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921779    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 18:37:21.229353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921781    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921784    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921787    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921790    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921792    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921796    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921798    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921815    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921818    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921821    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921825    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921832    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921834    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921837    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921840    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921843    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921846    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921849    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921851    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921854    2583 flags.go:64] FLAG: --v="2"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921858    2583 flags.go:64] FLAG: --version="false"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921862    2583 flags.go:64] FLAG: --vmodule=""
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921866    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.921869    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921973    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.230343 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921976    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921979    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921982    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921985    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921988    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921991    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921993    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921996    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.921998    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922001    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922004    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922006    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922009    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922011    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922014    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922016    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922019    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922022    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922026    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922028    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.231048 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922031    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922033    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922036    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922039    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922041    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922044    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922046    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922049    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922052    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922055    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922057    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922060    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922062    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922064    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922067    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922069    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922072    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922074    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922078    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.368280 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922082    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922085    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922088    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922090    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922093    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922096    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922098    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922101    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922103    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922106    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922109    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922115    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922119    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922123    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922126    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922129    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922131    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922134    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922137    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.368846 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922140    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922143    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922145    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922148    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922151    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922153    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922156    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922159    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922161    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922163    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922166    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922168    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922171    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922174    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922176    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922179    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922181    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922184    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922186    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922189    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.369347 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922191    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922194    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922196    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922200    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922204    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922207    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922210    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922212    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922215    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922217    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922220    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922222    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922225    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922228    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922230    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922233    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922236    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922239    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922241    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.369917 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.922244    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.922789    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.930220    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.930349    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.930371    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.930379    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930464    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930470    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930474    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930479    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930483    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930486    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930489    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930493    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930496    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.370415 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930499    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930503    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930506    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930510    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930513    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930516    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930520    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930524    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930529    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930533    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930536    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930539    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930543    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930546    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930549    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930559    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930563    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930566    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930569    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.370880 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930573    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930576    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930580    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930583    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930586    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930589    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930592    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930596    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930599    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930602    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930605    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930609    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930612    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930615    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930619    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930622    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930625    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930629    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930632    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930635    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.371380 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930642    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930646    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930652    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930657    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930662    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930667    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930670    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930674    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930678    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930690    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930693    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930697    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930701    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930705    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930708    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930712    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930716    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930720    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930724    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930728    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.371948 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930732    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930736    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930739    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930743    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930747    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930751    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930755    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930759    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930763    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930767    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930770    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930774    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930778    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930782    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930785    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930790    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930793    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930797    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930801    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.372482 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930818    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930822    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930827    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930838    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930842    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930846    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930849    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930854    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930857    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930861    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930865    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.930869    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.930877    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931077    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:21.373019 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931085    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931093    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931097    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931101    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931105    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931111    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931115    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931119    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931122    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931126    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931130    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931134    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931139    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931144    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931148    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931153    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931157    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931160    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931164    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:21.373397 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931168    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931172    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931176    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931187    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931192    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931195    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931199    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931203    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931206    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931210    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931214    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931217    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931221    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931225    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931230    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931234    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931238    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931242    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931247    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931251    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:21.373938 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931255    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931260    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931264    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931269    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931273    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931278    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931282    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931285    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931289    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931293    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931298    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931310    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931314    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931318    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931322    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931326    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931337    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931341    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931345    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:21.374488 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931349    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931353    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931356    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931360    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931364    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931377    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931382    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931390    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931394    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931398    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931401    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931407    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931410    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931414    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931418    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931421    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931425    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931428    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931432    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931435    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:21.375035 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931439    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931443    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931446    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931449    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931453    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931456    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931460    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931464    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931467    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931471    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931482    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931485    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931489    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931493    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931496    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931500    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931504    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931508    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931511    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:21.375628 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:20.931517    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.931524    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.931617    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.934636    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.935858    2583 server.go:1019] "Starting client certificate rotation"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.935968    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.936876    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.938494    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.940034    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.952109    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.955557    2583 log.go:25] "Validated CRI v1 image API"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.956324    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 18:37:21.376196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.958242    2583 fs.go:135] Filesystem UUIDs: map[630aa113-4ae5-417b-bf46-d52dd06c56e0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a2c3056a-3e43-4785-9c0d-9a4226059a97:/dev/nvme0n1p4]
Jul 01 18:37:21.376543 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.958255    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.963527    2583 manager.go:216] Machine: {Timestamp:2026-07-01 18:37:20.962494074 +0000 UTC m=+0.280553101 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099789 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2472b265b676c54f1fdfc3f1e03a SystemUUID:ec2a2472-b265-b676-c54f-1fdfc3f1e03a BootID:b6405f4b-1077-43c6-9ab3-07721c754018 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:30:52:f5:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:30:52:f5:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:65:5e:aa:c4:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.963626    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.963709    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.966147    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.966173    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.966853    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.966863    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 18:37:21.376645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.966870    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.967466    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.968186    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971363    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971583    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971611    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971620    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971639    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971648    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971660    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971972    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.971990    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.973636    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.973859    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975152    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975184    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975757    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975775    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975781    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975788    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975796    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975814    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975822    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975827    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975833    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975839    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975869    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.975877    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.976558    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.976566    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.979665    2583 server.go:1268] "Started kubelet"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.979743    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.979840    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.980133    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.981079    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:20.985392    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:20.985738    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.985924    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:20.986048    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 18:37:21.376929 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.986395    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.989407    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.989424    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:20.989631    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990033    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990033    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990063    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990139    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990149    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990190    2583 factory.go:55] Registering systemd factory
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990204    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990402    2583 factory.go:153] Registering CRI-O factory
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990412    2583 factory.go:223] Registration of the crio container factory successfully
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990455    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990470    2583 factory.go:103] Registering Raw factory
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.990480    2583 manager.go:1196] Started watching for new ooms in manager
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:20.990194    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55dc9be05e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:20.979640414 +0000 UTC m=+0.297699423,LastTimestamp:2026-07-01 18:37:20.979640414 +0000 UTC m=+0.297699423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:20.991391    2583 manager.go:349] Starting recovery of all containers
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.005045    2583 manager.go:354] Recovery completed
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.005788    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.005930    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.011239    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.012956    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.012990    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.013002    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.013480    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.013489    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.013505    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.013548    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014661    2583 policy_none.go:49] "None policy: Start"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014676    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014688    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014718    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014741    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.377991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.014752    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.015258    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98860d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.012975117 +0000 UTC m=+0.331034127,LastTimestamp:2026-07-01 18:37:21.012975117 +0000 UTC m=+0.331034127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.015639    2583 policy_none.go:47] "Start"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.029274    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98da7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.012996731 +0000 UTC m=+0.331055742,LastTimestamp:2026-07-01 18:37:21.012996731 +0000 UTC m=+0.331055742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.046218    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98ff81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.013006209 +0000 UTC m=+0.331065219,LastTimestamp:2026-07-01 18:37:21.013006209 +0000 UTC m=+0.331065219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.057059    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.058176    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.058207    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.058224    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.058176    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-241.ec2.internal.18be3e55de98860d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98860d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.012975117 +0000 UTC m=+0.331034127,LastTimestamp:2026-07-01 18:37:21.014727365 +0000 UTC m=+0.332786371,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.058257    2583 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061441    2583 manager.go:337] "Starting Device Plugin manager"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.061471    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 18:37:21.379157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061480    2583 server.go:87] "Starting device plugin registration server"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061675    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061688    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061751    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061838    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.061848    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.062241    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.062287    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.080141    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-241.ec2.internal.18be3e55de98da7b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98da7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.012996731 +0000 UTC m=+0.331055742,LastTimestamp:2026-07-01 18:37:21.014746783 +0000 UTC m=+0.332805793,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.080230    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.092887    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-241.ec2.internal.18be3e55de98ff81\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55de98ff81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.013006209 +0000 UTC m=+0.331065219,LastTimestamp:2026-07-01 18:37:21.014756093 +0000 UTC m=+0.332815101,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.093054    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7n7v"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.101196    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7n7v"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.102407    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18be3e55e19ceb95  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-07-01 18:37:21.063594901 +0000 UTC m=+0.381653899,LastTimestamp:2026-07-01 18:37:21.063594901 +0000 UTC m=+0.381653899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.158518    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"]
Jul 01 18:37:21.529191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.158567    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.159888    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.159917    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.159927    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161225    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161367    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161778    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161921    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161935    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161957    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161960    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161970    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.161972    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.162517    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.162540    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.162553    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.162577    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.163218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.163240    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.163911    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.163938    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.163952    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.180724    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.180740    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-241.ec2.internal\": node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.181297    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-241.ec2.internal\" not found" node="ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.184619    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-241.ec2.internal\" not found" node="ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.214992    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.291352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.532944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.291378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.291397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08fe03195acc53e90bd3f1c6ffa3dc8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"e08fe03195acc53e90bd3f1c6ffa3dc8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.315657    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08fe03195acc53e90bd3f1c6ffa3dc8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"e08fe03195acc53e90bd3f1c6ffa3dc8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392383    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cea8b0605e3ff78b65b82246c05dcc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"0cea8b0605e3ff78b65b82246c05dcc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.392423    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08fe03195acc53e90bd3f1c6ffa3dc8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"e08fe03195acc53e90bd3f1c6ffa3dc8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.415856    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.484310    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.488243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:21.533733 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.516430    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.695136 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.616904    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.869785 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.717335    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.869785 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.817823    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Jul 01 18:37:21.986283 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.894399    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:21.986283 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.935654    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 18:37:21.986283 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.935776    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:37:21.986283 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.971905    2583 apiserver.go:52] "Watching apiserver"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.987857    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.989618    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.989697    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.991904    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mrwgc","openshift-multus/multus-additional-cni-plugins-2mg9d","openshift-multus/network-metrics-daemon-4pb8p","openshift-network-diagnostics/network-check-target-2kb8s","openshift-ovn-kubernetes/ovnkube-node-bpxvb","kube-system/global-pull-secret-syncer-t5z88","kube-system/konnectivity-agent-spjhz","openshift-cluster-node-tuning-operator/tuned-zqtht","openshift-multus/multus-szrhw","openshift-network-operator/iptables-alerter-74269","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv","openshift-dns/node-resolver-64k9d"]
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.994137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.995425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.996573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.996650    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997149    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997150    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sfmjd\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997226    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997540    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997733    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997766    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.997819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:21.997881    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.998035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfcm8\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.998057    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:21.999273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.000662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:22.000715    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:23.402708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001395    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001737    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.001987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.002683    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.002693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-92xp6\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.003107    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.003160    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.003325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.004177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.004454    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.004641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.004767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qp2bb\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.005867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.006256    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.006451    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nv7jh\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.006451    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.006782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xnt2k\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.007009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.007011    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.007284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.007831    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.008233    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.008241    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.008244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b5fps\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.008467    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"]
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.008528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.009382    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.009492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gx5nd\""
Jul 01 18:37:23.404184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.009523    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.009524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.011136    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8px5\""
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.011161    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:22.011244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.318594    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.318725    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"]
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.320275    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cea8b0605e3ff78b65b82246c05dcc8.slice/crio-d1990a44234843dae4c064e9143855d7fb1195d99eb6d15f04831beaf57a1ac5 WatchSource:0}: Error finding container d1990a44234843dae4c064e9143855d7fb1195d99eb6d15f04831beaf57a1ac5: Status 404 returned error can't find the container with id d1990a44234843dae4c064e9143855d7fb1195d99eb6d15f04831beaf57a1ac5
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.320487    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08fe03195acc53e90bd3f1c6ffa3dc8.slice/crio-8f5c0761de2836690641f38d49f726b6c0a00161961f912fec70c454efcd8934 WatchSource:0}: Error finding container 8f5c0761de2836690641f38d49f726b6c0a00161961f912fec70c454efcd8934: Status 404 returned error can't find the container with id 8f5c0761de2836690641f38d49f726b6c0a00161961f912fec70c454efcd8934
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.325703    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.364720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"0cea8b0605e3ff78b65b82246c05dcc8","Type":"ContainerStarted","Data":"d1990a44234843dae4c064e9143855d7fb1195d99eb6d15f04831beaf57a1ac5"}
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.365483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" event={"ID":"e08fe03195acc53e90bd3f1c6ffa3dc8","Type":"ContainerStarted","Data":"8f5c0761de2836690641f38d49f726b6c0a00161961f912fec70c454efcd8934"}
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.374348    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zsxdq"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.393710    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.394883    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zsxdq"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.398955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-netns\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.399167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-serviceca\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.399519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-systemd-units\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.399670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-log-socket\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.405155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.399800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-cnibin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-multus-certs\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-registration-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-system-cni-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-sys\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-bin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-multus\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-socket-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d52c436-737c-49d3-bf32-a3c80dec20e1-hosts-file\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-kubelet-config\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-lib-modules\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-sys-fs\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.405905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-hostroot\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-daemon-config\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-etc-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-netns\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-script-lib\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvx9\" (UniqueName: \"kubernetes.io/projected/6b0835ee-0f6a-477c-84e3-e71b5469fb24-kube-api-access-lsvx9\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-device-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-etc-selinux\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhh6g\" (UniqueName: \"kubernetes.io/projected/64767e71-15b6-4f29-8987-4c19f54f6f41-kube-api-access-mhh6g\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0835ee-0f6a-477c-84e3-e71b5469fb24-host-slash\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.406414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-dbus\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-modprobe-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/8a63e83e-5bc8-417d-9d12-ccb369d53076-kube-api-access-t9vh4\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-tuned\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-conf-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-kube-api-access-98wlv\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4p9\" (UniqueName: \"kubernetes.io/projected/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-kube-api-access-fs4p9\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-netd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-kubernetes\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-systemd\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-etc-kubernetes\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.406910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-os-release\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ae8699-9280-465a-a248-6eade14246da-konnectivity-ca\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-tmp\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.400989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-ovn\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-node-log\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysconfig\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-os-release\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-cni-binary-copy\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-env-overrides\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-run\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-socket-dir-parent\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-cnibin\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.407699 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85t5k\" (UniqueName: \"kubernetes.io/projected/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-kube-api-access-85t5k\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-kubelet\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ae8699-9280-465a-a248-6eade14246da-agent-certs\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tbs\" (UniqueName: \"kubernetes.io/projected/0d52c436-737c-49d3-bf32-a3c80dec20e1-kube-api-access-g7tbs\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-slash\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-systemd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-bin\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-config\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-conf\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-host\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-kubelet\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.408227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-var-lib-kubelet\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-host\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545n4\" (UniqueName: \"kubernetes.io/projected/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-kube-api-access-545n4\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-k8s-cni-cncf-io\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7pp\" (UniqueName: \"kubernetes.io/projected/113e4450-50df-42f3-b6d5-44b62ed68b61-kube-api-access-fp7pp\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b0835ee-0f6a-477c-84e3-e71b5469fb24-iptables-alerter-script\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d52c436-737c-49d3-bf32-a3c80dec20e1-tmp-dir\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-var-lib-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.401599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-system-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.407267    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 18:32:22 +0000 UTC" deadline="2027-11-27 17:21:42.805989109 +0000 UTC"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.407305    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12334h44m19.398687319s"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.434701    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-serviceca\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-systemd-units\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.537944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-log-socket\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-cnibin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-multus-certs\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-registration-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502160    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-systemd-units\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-system-cni-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502193    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-log-socket\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502213    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-cnibin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502221    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-registration-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-sys\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502253    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-system-cni-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-bin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502255    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-multus\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502300    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-multus-certs\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502320    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-sys\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502339    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-bin\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502369    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-cni-multus\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-socket-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d52c436-737c-49d3-bf32-a3c80dec20e1-hosts-file\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-kubelet-config\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502477    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d52c436-737c-49d3-bf32-a3c80dec20e1-hosts-file\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502477    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-kubelet-config\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502497    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-socket-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-lib-modules\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-sys-fs\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502575    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-serviceca\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-sys-fs\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.538944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.502672    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-hostroot\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-hostroot\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-daemon-config\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503250    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-etc-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-netns\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503314    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-etc-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503354    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503356    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-netns\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.503370    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-script-lib\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539441 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.503426    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.003408273 +0000 UTC m=+2.018987818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvx9\" (UniqueName: \"kubernetes.io/projected/6b0835ee-0f6a-477c-84e3-e71b5469fb24-kube-api-access-lsvx9\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-device-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-etc-selinux\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhh6g\" (UniqueName: \"kubernetes.io/projected/64767e71-15b6-4f29-8987-4c19f54f6f41-kube-api-access-mhh6g\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0835ee-0f6a-477c-84e3-e71b5469fb24-host-slash\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-dbus\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-modprobe-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/8a63e83e-5bc8-417d-9d12-ccb369d53076-kube-api-access-t9vh4\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503732    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-daemon-config\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-tuned\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-conf-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.539884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-kube-api-access-98wlv\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4p9\" (UniqueName: \"kubernetes.io/projected/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-kube-api-access-fs4p9\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-netd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-kubernetes\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-systemd\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503937    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-lib-modules\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-etc-kubernetes\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503954    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-script-lib\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503984    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-conf-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-os-release\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504075    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-device-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504079    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504092    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-modprobe-d\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540366 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504101    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504131    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-etc-selinux\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504145    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-systemd\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504154    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-os-release\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ae8699-9280-465a-a248-6eade14246da-konnectivity-ca\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-tmp\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-ovn\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504259    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b0835ee-0f6a-477c-84e3-e71b5469fb24-host-slash\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-node-log\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.503972    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504316    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504324    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-netd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysconfig\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-os-release\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.540842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-cni-binary-copy\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048bfc0b-24ff-4316-8586-8a828d189524-dbus\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-kubernetes\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-env-overrides\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-run\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-socket-dir-parent\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-cnibin\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-etc-kubernetes\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85t5k\" (UniqueName: \"kubernetes.io/projected/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-kube-api-access-85t5k\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-kubelet\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.504616    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ae8699-9280-465a-a248-6eade14246da-agent-certs\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64767e71-15b6-4f29-8987-4c19f54f6f41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.541376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504678    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-cnibin\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.504687    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.004672522 +0000 UTC m=+2.020252043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504726    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-os-release\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tbs\" (UniqueName: \"kubernetes.io/projected/0d52c436-737c-49d3-bf32-a3c80dec20e1-kube-api-access-g7tbs\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504786    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-env-overrides\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-slash\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-systemd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504854    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-systemd\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.504887    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysconfig\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505025    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-run\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-multus-socket-dir-parent\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505086    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/113e4450-50df-42f3-b6d5-44b62ed68b61-cni-binary-copy\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-bin\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.541853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505128    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-var-lib-kubelet\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-config\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-conf\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-host\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-kubelet\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-var-lib-kubelet\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-host\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545n4\" (UniqueName: \"kubernetes.io/projected/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-kube-api-access-545n4\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505363    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ae8699-9280-465a-a248-6eade14246da-konnectivity-ca\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-k8s-cni-cncf-io\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7pp\" (UniqueName: \"kubernetes.io/projected/113e4450-50df-42f3-b6d5-44b62ed68b61-kube-api-access-fp7pp\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505415    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-node-log\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b0835ee-0f6a-477c-84e3-e71b5469fb24-iptables-alerter-script\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.542352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505450    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-run-ovn\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d52c436-737c-49d3-bf32-a3c80dec20e1-tmp-dir\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505516    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505740    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-host\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505868    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a63e83e-5bc8-417d-9d12-ccb369d53076-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505901    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-cni-bin\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505942    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63e83e-5bc8-417d-9d12-ccb369d53076-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.505986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-var-lib-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-system-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-netns\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506128    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-netns\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506261    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovnkube-config\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506360    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-sysctl-conf\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506393    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-host\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.542770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506423    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-kubelet\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506464    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-var-lib-kubelet\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d52c436-737c-49d3-bf32-a3c80dec20e1-tmp-dir\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-host-run-k8s-cni-cncf-io\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b0835ee-0f6a-477c-84e3-e71b5469fb24-iptables-alerter-script\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506939    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-var-lib-openvswitch\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.506967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-host-slash\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.507003    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/113e4450-50df-42f3-b6d5-44b62ed68b61-system-cni-dir\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.508148    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-etc-tuned\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.508299    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-tmp\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.508382    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.508441    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ae8699-9280-465a-a248-6eade14246da-agent-certs\") pod \"konnectivity-agent-spjhz\" (UID: \"53ae8699-9280-465a-a248-6eade14246da\") " pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.525341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85t5k\" (UniqueName: \"kubernetes.io/projected/18b8b9ab-f5d2-4f95-9be8-a0237c6c9533-kube-api-access-85t5k\") pod \"ovnkube-node-bpxvb\" (UID: \"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.532180    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.532201    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:23.543240 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.532215    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:23.532273    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.03225751 +0000 UTC m=+2.047837047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.533099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7pp\" (UniqueName: \"kubernetes.io/projected/113e4450-50df-42f3-b6d5-44b62ed68b61-kube-api-access-fp7pp\") pod \"multus-szrhw\" (UID: \"113e4450-50df-42f3-b6d5-44b62ed68b61\") " pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.533228    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tbs\" (UniqueName: \"kubernetes.io/projected/0d52c436-737c-49d3-bf32-a3c80dec20e1-kube-api-access-g7tbs\") pod \"node-resolver-64k9d\" (UID: \"0d52c436-737c-49d3-bf32-a3c80dec20e1\") " pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.533826    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhh6g\" (UniqueName: \"kubernetes.io/projected/64767e71-15b6-4f29-8987-4c19f54f6f41-kube-api-access-mhh6g\") pod \"aws-ebs-csi-driver-node-fprbv\" (UID: \"64767e71-15b6-4f29-8987-4c19f54f6f41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.533842    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/8a63e83e-5bc8-417d-9d12-ccb369d53076-kube-api-access-t9vh4\") pod \"multus-additional-cni-plugins-2mg9d\" (UID: \"8a63e83e-5bc8-417d-9d12-ccb369d53076\") " pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.534068    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545n4\" (UniqueName: \"kubernetes.io/projected/9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6-kube-api-access-545n4\") pod \"tuned-zqtht\" (UID: \"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6\") " pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.534521    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/ef1f2162-8c0c-42df-87fb-ae2da76b0cd0-kube-api-access-98wlv\") pod \"node-ca-mrwgc\" (UID: \"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0\") " pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.534793    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvx9\" (UniqueName: \"kubernetes.io/projected/6b0835ee-0f6a-477c-84e3-e71b5469fb24-kube-api-access-lsvx9\") pod \"iptables-alerter-74269\" (UID: \"6b0835ee-0f6a-477c-84e3-e71b5469fb24\") " pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.543697 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.534916    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4p9\" (UniqueName: \"kubernetes.io/projected/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-kube-api-access-fs4p9\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.629925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mrwgc"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.631569    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.639563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mg9d"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.643142    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1f2162_8c0c_42df_87fb_ae2da76b0cd0.slice/crio-ba461c88accdc8cb6156a28f622233b0c3238194d283195e48bb5755829a0001 WatchSource:0}: Error finding container ba461c88accdc8cb6156a28f622233b0c3238194d283195e48bb5755829a0001: Status 404 returned error can't find the container with id ba461c88accdc8cb6156a28f622233b0c3238194d283195e48bb5755829a0001
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.647004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.650928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.657055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqtht"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.660429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-szrhw"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.660437    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b8b9ab_f5d2_4f95_9be8_a0237c6c9533.slice/crio-f9c81430c8d12451d919b92f10302472264ad98c932cb644a2c1e0eb104fb68c WatchSource:0}: Error finding container f9c81430c8d12451d919b92f10302472264ad98c932cb644a2c1e0eb104fb68c: Status 404 returned error can't find the container with id f9c81430c8d12451d919b92f10302472264ad98c932cb644a2c1e0eb104fb68c
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.661110    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ae8699_9280_465a_a248_6eade14246da.slice/crio-00bbfe44345d68689c987bf0180e0c8d4a359c014109d8cea42b8f302639c4c6 WatchSource:0}: Error finding container 00bbfe44345d68689c987bf0180e0c8d4a359c014109d8cea42b8f302639c4c6: Status 404 returned error can't find the container with id 00bbfe44345d68689c987bf0180e0c8d4a359c014109d8cea42b8f302639c4c6
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.665967    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed563e8_8b20_4e0b_a8ef_8ae0b0b985b6.slice/crio-99aa28c2e975818693fd129c3d6da97d4988d3d499a27718fc9a9175b2648e7a WatchSource:0}: Error finding container 99aa28c2e975818693fd129c3d6da97d4988d3d499a27718fc9a9175b2648e7a: Status 404 returned error can't find the container with id 99aa28c2e975818693fd129c3d6da97d4988d3d499a27718fc9a9175b2648e7a
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.666918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-74269"
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.668821    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113e4450_50df_42f3_b6d5_44b62ed68b61.slice/crio-b8c904b1da4505dd50cfc6b6268c5aa59713686266ddb31d01b711975f23ccba WatchSource:0}: Error finding container b8c904b1da4505dd50cfc6b6268c5aa59713686266ddb31d01b711975f23ccba: Status 404 returned error can't find the container with id b8c904b1da4505dd50cfc6b6268c5aa59713686266ddb31d01b711975f23ccba
Jul 01 18:37:23.675108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.675095    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv"
Jul 01 18:37:23.680754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.679965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64k9d"
Jul 01 18:37:23.682617 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:23.682190    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.785319 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:23.690222    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0835ee_0f6a_477c_84e3_e71b5469fb24.slice/crio-3b3f5e769b9965f2bc88eb8bef535488beeb0c05cec6d1e572412387f19b8907 WatchSource:0}: Error finding container 3b3f5e769b9965f2bc88eb8bef535488beeb0c05cec6d1e572412387f19b8907: Status 404 returned error can't find the container with id 3b3f5e769b9965f2bc88eb8bef535488beeb0c05cec6d1e572412387f19b8907
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.009448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.009514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.009613    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.009659    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.009678    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.009660032 +0000 UTC m=+3.025239564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.010280 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.009710    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.00969308 +0000 UTC m=+3.025272602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.183869 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.110098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:24.183869 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.110308    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:24.183869 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.110362    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:24.183869 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.110375    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:24.183869 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:24.110427    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.110411076 +0000 UTC m=+3.125990612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:24.397393 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.397305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64k9d" event={"ID":"0d52c436-737c-49d3-bf32-a3c80dec20e1","Type":"ContainerStarted","Data":"8daaaa710c2c8aa24c768fc90e1fc31f5be3243eda7f5038f5e05a4cdec9d02b"}
Jul 01 18:37:24.398013 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.397439    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:23 +0000 UTC" deadline="2027-12-11 01:24:29.166036357 +0000 UTC"
Jul 01 18:37:24.398013 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.397468    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12654h47m4.768571762s"
Jul 01 18:37:24.417336 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.417302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-74269" event={"ID":"6b0835ee-0f6a-477c-84e3-e71b5469fb24","Type":"ContainerStarted","Data":"3b3f5e769b9965f2bc88eb8bef535488beeb0c05cec6d1e572412387f19b8907"}
Jul 01 18:37:24.421977 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.421949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-szrhw" event={"ID":"113e4450-50df-42f3-b6d5-44b62ed68b61","Type":"ContainerStarted","Data":"b8c904b1da4505dd50cfc6b6268c5aa59713686266ddb31d01b711975f23ccba"}
Jul 01 18:37:24.438268 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.438215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqtht" event={"ID":"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6","Type":"ContainerStarted","Data":"99aa28c2e975818693fd129c3d6da97d4988d3d499a27718fc9a9175b2648e7a"}
Jul 01 18:37:24.455901 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.455822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spjhz" event={"ID":"53ae8699-9280-465a-a248-6eade14246da","Type":"ContainerStarted","Data":"00bbfe44345d68689c987bf0180e0c8d4a359c014109d8cea42b8f302639c4c6"}
Jul 01 18:37:24.469569 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.469409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"f9c81430c8d12451d919b92f10302472264ad98c932cb644a2c1e0eb104fb68c"}
Jul 01 18:37:24.488469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.488366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv" event={"ID":"64767e71-15b6-4f29-8987-4c19f54f6f41","Type":"ContainerStarted","Data":"51b4b43b5bdfd79aef03d20eb04c9930959eceb5d60a115745f57a1d38e81a13"}
Jul 01 18:37:24.505830 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.505737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerStarted","Data":"ba09c5e9ffd2fc398977aa8f946cca75968c22ffd8d54cbf2301d09cfb4ca1b5"}
Jul 01 18:37:24.518773 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:24.518744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mrwgc" event={"ID":"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0","Type":"ContainerStarted","Data":"ba461c88accdc8cb6156a28f622233b0c3238194d283195e48bb5755829a0001"}
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.020516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.020595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.020708    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.020761    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.020742978 +0000 UTC m=+5.036322502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.020865    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:25.021138 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.020925    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.02090788 +0000 UTC m=+5.036487404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:25.121513 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.121480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:25.121673 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.121630    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:25.121673 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.121653    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:25.121673 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.121665    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:25.121830 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.121716    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.121698761 +0000 UTC m=+5.137278286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:25.361732 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.361657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:25.361886 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.361790    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:25.362218 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.362198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:25.362602 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.362314    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:25.362602 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.362393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:25.362602 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:25.362492    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:25.398685 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.398604    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:23 +0000 UTC" deadline="2028-02-29 12:37:53.752138849 +0000 UTC"
Jul 01 18:37:25.398685 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:25.398651    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14586h0m28.35349145s"
Jul 01 18:37:26.027982 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:26.027953    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:27.037384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.037356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:27.037814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.037412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:27.037814 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.037548    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:27.037814 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.037606    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.037588294 +0000 UTC m=+9.053167818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:27.037814 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.037549    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:27.037814 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.037651    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.037638495 +0000 UTC m=+9.053218022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:27.138075 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.138042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:27.138230 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.138217    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:27.138304 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.138238    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:27.138304 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.138251    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:27.138403 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.138322    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.138303604 +0000 UTC m=+9.153883130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:27.361262 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.361189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:27.361422 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.361330    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:27.361666 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.361631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:27.361783 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:27.361682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:27.361783 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.361743    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:27.361911 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:27.361822    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:29.360948 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:29.360916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:29.361504 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:29.361062    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:29.361504 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:29.361444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:29.361613 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:29.361547    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:29.361613 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:29.361605    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:29.361721 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:29.361703    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:31.070041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.070002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:31.070457 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.070067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:31.070457 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.070178    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:31.070457 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.070239    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.070224089 +0000 UTC m=+17.085803610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:31.070457 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.070178    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.070457 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.070353    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.07033385 +0000 UTC m=+17.085913381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.170788 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.170756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:31.170941 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.170928    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:31.171004 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.170953    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:31.171004 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.170965    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:31.171108 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.171019    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.171001346 +0000 UTC m=+17.186580875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:31.361020 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.360902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:31.361020 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.360905    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:31.361228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.361018    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:31.361228 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:31.360913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:31.361228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.361110    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:31.361228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:31.361164    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:33.360917 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:33.360888    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:33.360917 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:33.360912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:33.361550 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:33.361006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:33.361550 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:33.361349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:33.361550 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:33.361445    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:33.361550 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:33.361531    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:35.361534 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:35.361503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:35.361946 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:35.361503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:35.361946 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:35.361633    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:35.361946 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:35.361502    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:35.361946 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:35.361736    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:35.361946 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:35.361787    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:37.361248 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:37.361214    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:37.361667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:37.361373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:37.361667 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:37.361389    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:37.361667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:37.361409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:37.361667 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:37.361487    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:37.361667 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:37.361564    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:39.131728 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.131694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:39.132103 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.131736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:39.132103 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.131819    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.132103 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.131824    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:39.132103 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.131875    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:55.131858305 +0000 UTC m=+33.147437829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.132103 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.131895    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:55.131884935 +0000 UTC m=+33.147464463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:39.232124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.232096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:39.232306 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.232229    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:39.232306 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.232249    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:39.232306 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.232261    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:39.232467 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.232321    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:55.232307197 +0000 UTC m=+33.247886723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:39.361004 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.360937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:39.361160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.361046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:39.361160 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.361052    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:39.361160 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.361145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:39.361338 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:39.361053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:39.361338 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:39.361264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:41.361199 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:41.361171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:41.361638 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:41.361173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:41.361638 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:41.361265    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:41.361638 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:41.361390    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:41.361638 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:41.361177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:41.361638 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:41.361484    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:43.361326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.361168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:43.361688 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.361212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:43.361688 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:43.361447    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:43.361688 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.361213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:43.361688 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:43.361483    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:43.361688 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:43.361540    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:43.568753 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.568686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-szrhw" event={"ID":"113e4450-50df-42f3-b6d5-44b62ed68b61","Type":"ContainerStarted","Data":"980296c9e3e349f0d74bc03acca602afd02cb4663bfd2a19acb05fa83ace4884"}
Jul 01 18:37:43.575738 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.575705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqtht" event={"ID":"9ed563e8-8b20-4e0b-a8ef-8ae0b0b985b6","Type":"ContainerStarted","Data":"3ef2c019ce0086c044fa5115a45edd1e88e3194bdd90467f241c8b2e6de52a6a"}
Jul 01 18:37:43.577489 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.577464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" event={"ID":"e08fe03195acc53e90bd3f1c6ffa3dc8","Type":"ContainerStarted","Data":"d261c8ea5f8358a2da8b62b322015e73f0607a99566b1bf659b484db161843e3"}
Jul 01 18:37:43.592519 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.592476    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-szrhw" podStartSLOduration=2.217755898 podStartE2EDuration="21.592463056s" totalImagesPullingTime="19.374707158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:43.591929961 +0000 UTC m=+21.607509505" watchObservedRunningTime="2026-07-01 18:37:43.592463056 +0000 UTC m=+21.608042599"
Jul 01 18:37:43.647181 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:43.646712    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" podStartSLOduration=20.646690797 podStartE2EDuration="20.646690797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:43.617939766 +0000 UTC m=+21.633519310" watchObservedRunningTime="2026-07-01 18:37:43.646690797 +0000 UTC m=+21.662270342"
Jul 01 18:37:44.580614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.580408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spjhz" event={"ID":"53ae8699-9280-465a-a248-6eade14246da","Type":"ContainerStarted","Data":"53041328811ec2843f78af8607e0ff6631a577f55a018cda5a09502d300939e9"}
Jul 01 18:37:44.583323 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583304    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:37:44.583644 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583627    2583 generic.go:356] "Generic (PLEG): container finished" podID="18b8b9ab-f5d2-4f95-9be8-a0237c6c9533" containerID="1d3cbf506413bccead725f848190787b12dc98a11872d7bd4631fca2b68cf8c6" exitCode=1
Jul 01 18:37:44.583716 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"53bb4e91456ecce916c8bf9d5b541e8fac515969f20d057eeb028953a37e55e4"}
Jul 01 18:37:44.583780 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"c4fe0f3f947725eb55b6b8a5a083cf2a1d1a749b26f5808c141ea5c9d9cb2973"}
Jul 01 18:37:44.583839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"3ab41ee7fd37a459472849af098c5e22079998fd156d90f469f22559ca43f5af"}
Jul 01 18:37:44.583889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"3f775f23eeb664cd3b209a4f6d83fecd114cdbe7e1ebdf3fd4b8f787b45da826"}
Jul 01 18:37:44.583889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerDied","Data":"1d3cbf506413bccead725f848190787b12dc98a11872d7bd4631fca2b68cf8c6"}
Jul 01 18:37:44.583979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.583928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"b0efcd5cf51a6d48b0273c136cae1b166da65a2780b33cbf3c1a5fc7cab23c79"}
Jul 01 18:37:44.585188 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.585166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv" event={"ID":"64767e71-15b6-4f29-8987-4c19f54f6f41","Type":"ContainerStarted","Data":"2ac1b5f07c34d1000e30d46d504720fbaf67df8515ce6370bdb677e3023b7e0e"}
Jul 01 18:37:44.586750 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.586729    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="b46c4e383d93c5511dea80ed4c6e31bc40b09cc60db1ccae41bde6e627ad92fc" exitCode=0
Jul 01 18:37:44.586846 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.586804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"b46c4e383d93c5511dea80ed4c6e31bc40b09cc60db1ccae41bde6e627ad92fc"}
Jul 01 18:37:44.588232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.588216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mrwgc" event={"ID":"ef1f2162-8c0c-42df-87fb-ae2da76b0cd0","Type":"ContainerStarted","Data":"1e6c7aba87d4a5f229f1824ac85db1bf697045e5deebace61490634794addf96"}
Jul 01 18:37:44.589673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.589656    2583 generic.go:356] "Generic (PLEG): container finished" podID="0cea8b0605e3ff78b65b82246c05dcc8" containerID="1d97913772645af9eefff06946b9ee791a98a30dd2cba1fea751e5d6c0b07de9" exitCode=0
Jul 01 18:37:44.589738 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.589687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"0cea8b0605e3ff78b65b82246c05dcc8","Type":"ContainerDied","Data":"1d97913772645af9eefff06946b9ee791a98a30dd2cba1fea751e5d6c0b07de9"}
Jul 01 18:37:44.591156 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.591132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64k9d" event={"ID":"0d52c436-737c-49d3-bf32-a3c80dec20e1","Type":"ContainerStarted","Data":"ab642031f98fd4caea3fd801213f2f348d6462e71054052bb5a3d70ab09629f5"}
Jul 01 18:37:44.592993 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.592950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-74269" event={"ID":"6b0835ee-0f6a-477c-84e3-e71b5469fb24","Type":"ContainerStarted","Data":"033dcc3e71cc6b3a78af81a2052980197848f388bfd428132ba3513992c4652d"}
Jul 01 18:37:44.599657 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.599622    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zqtht" podStartSLOduration=3.291104736 podStartE2EDuration="22.599609772s" totalImagesPullingTime="19.308505036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:43.646638481 +0000 UTC m=+21.662218026" watchObservedRunningTime="2026-07-01 18:37:44.599609772 +0000 UTC m=+22.615189332"
Jul 01 18:37:44.620842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.620798    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-spjhz" podStartSLOduration=3.331058551 podStartE2EDuration="22.620788456s" totalImagesPullingTime="19.289729905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.599791682 +0000 UTC m=+22.615371226" watchObservedRunningTime="2026-07-01 18:37:44.620788456 +0000 UTC m=+22.636368001"
Jul 01 18:37:44.621055 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.621031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-74269" podStartSLOduration=3.336486771 podStartE2EDuration="22.621024685s" totalImagesPullingTime="19.284537914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.620705698 +0000 UTC m=+22.636285234" watchObservedRunningTime="2026-07-01 18:37:44.621024685 +0000 UTC m=+22.636604230"
Jul 01 18:37:44.639106 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.639067    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-64k9d" podStartSLOduration=7.34859599 podStartE2EDuration="22.639056551s" totalImagesPullingTime="15.290460561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.638754783 +0000 UTC m=+22.654334347" watchObservedRunningTime="2026-07-01 18:37:44.639056551 +0000 UTC m=+22.654636095"
Jul 01 18:37:44.713277 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:44.713115    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mrwgc" podStartSLOduration=3.381713608 podStartE2EDuration="22.713101628s" totalImagesPullingTime="19.33138802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.682035925 +0000 UTC m=+22.697615460" watchObservedRunningTime="2026-07-01 18:37:44.713101628 +0000 UTC m=+22.728681172"
Jul 01 18:37:45.079577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.079482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:45.080169 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.080152    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:45.361141 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.361071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:45.361311 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.361072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:45.361311 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:45.361174    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:45.361311 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.361078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:45.361311 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:45.361254    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:45.361522 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:45.361367    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:45.551973 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.551953    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 18:37:45.596673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.596638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"0cea8b0605e3ff78b65b82246c05dcc8","Type":"ContainerStarted","Data":"dc8e09780638fd289b88002073080e79e3a99af9e87fdc1339618232a51a8d72"}
Jul 01 18:37:45.598675 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.598634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv" event={"ID":"64767e71-15b6-4f29-8987-4c19f54f6f41","Type":"ContainerStarted","Data":"f39e4580731f6637395b36f2e55a831bb593d5b2113c3865fba69cef589d27eb"}
Jul 01 18:37:45.599155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.599134    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:45.599331 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.599316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-spjhz"
Jul 01 18:37:45.637602 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:45.637555    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" podStartSLOduration=22.637538949 podStartE2EDuration="22.637538949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:45.619134043 +0000 UTC m=+23.634713587" watchObservedRunningTime="2026-07-01 18:37:45.637538949 +0000 UTC m=+23.653118494"
Jul 01 18:37:46.380517 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:46.380425    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T18:37:45.551968549Z","UUID":"c10e7444-e84c-4305-abdf-6bb0ee78cc71","Handler":null,"Name":"","Endpoint":""}
Jul 01 18:37:46.382266 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:46.382245    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 18:37:46.382441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:46.382317    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 18:37:46.604791 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:46.604769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:37:46.605264 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:46.605235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"84af228057909bb68c863aa20803ab810d6e35338a3d976f45c4d91c59139999"}
Jul 01 18:37:47.361298 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:47.361253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:47.361298 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:47.361265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:47.361647 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:47.361381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:47.361647 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:47.361380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:47.361647 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:47.361499    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:47.361647 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:47.361586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:47.609481 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:47.609447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv" event={"ID":"64767e71-15b6-4f29-8987-4c19f54f6f41","Type":"ContainerStarted","Data":"5d60f7f275b6aa1bcdb61117903ae58a2c224909045629eb13272c18621d8e80"}
Jul 01 18:37:47.632184 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:47.632144    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fprbv" podStartSLOduration=4.284410184 podStartE2EDuration="25.632129331s" totalImagesPullingTime="21.347719147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:47.632105628 +0000 UTC m=+25.647685192" watchObservedRunningTime="2026-07-01 18:37:47.632129331 +0000 UTC m=+25.647708868"
Jul 01 18:37:49.362083 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.361805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:49.362611 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.361870    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:49.362611 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:49.362111    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:49.362611 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.361882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:49.362611 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:49.362218    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:49.362611 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:49.362302    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:49.617603 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.617581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:37:49.618012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.617989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"177c1c0c2478953fec6cb684042adb96244e40664f5705c065a1b2bda201c36c"}
Jul 01 18:37:49.618297 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.618258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:49.618434 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.618363    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:49.618434 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.618431    2583 scope.go:117] "RemoveContainer" containerID="1d3cbf506413bccead725f848190787b12dc98a11872d7bd4631fca2b68cf8c6"
Jul 01 18:37:49.620465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.620431    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="c3921fe93a0838551f43ac8bf396f99d7db9479be6865a235f97867035e62830" exitCode=0
Jul 01 18:37:49.620583 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.620492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"c3921fe93a0838551f43ac8bf396f99d7db9479be6865a235f97867035e62830"}
Jul 01 18:37:49.635822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:49.635797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:50.627047 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:50.627027    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:37:50.627808 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:50.627782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" event={"ID":"18b8b9ab-f5d2-4f95-9be8-a0237c6c9533","Type":"ContainerStarted","Data":"722e16381e5e9fa4fffb535c522a9fb87acdc62beba5fb02065b1ab5cbc02e5a"}
Jul 01 18:37:50.628020 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:50.628003    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:50.644481 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:50.644459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:37:50.674819 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:50.674773    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb" podStartSLOduration=8.715266481 podStartE2EDuration="28.674761002s" totalImagesPullingTime="19.959494521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:50.674228492 +0000 UTC m=+28.689808061" watchObservedRunningTime="2026-07-01 18:37:50.674761002 +0000 UTC m=+28.690340569"
Jul 01 18:37:51.053969 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.053760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kb8s"]
Jul 01 18:37:51.054105 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.054050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:51.054147 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:51.054130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:51.061133 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.061111    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4pb8p"]
Jul 01 18:37:51.061255 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.061225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:51.061372 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:51.061351    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:51.061862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.061826    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5z88"]
Jul 01 18:37:51.061930 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.061914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:51.062020 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:51.062001    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:51.632007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.631977    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="0a8fabb3ddc5b50491d68dd1272d7491f43738c88d029ab8f0d4d81e5870aff4" exitCode=0
Jul 01 18:37:51.632414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:51.632065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"0a8fabb3ddc5b50491d68dd1272d7491f43738c88d029ab8f0d4d81e5870aff4"}
Jul 01 18:37:52.361811 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:52.361784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:52.362017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:52.361901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:52.362017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:52.361927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:52.362017 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:52.361904    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:52.362174 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:52.362017    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:52.362174 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:52.362101    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:53.637707 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:53.637676    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="5085ab3c9a0580c3df29c88f1945852eca072793ac35a9664c56ca23ac1cc01f" exitCode=0
Jul 01 18:37:53.638201 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:53.637719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"5085ab3c9a0580c3df29c88f1945852eca072793ac35a9664c56ca23ac1cc01f"}
Jul 01 18:37:54.361085 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:54.361054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:54.361085 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:54.361077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:54.361085 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:54.361077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:54.361365 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:54.361191    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:54.361365 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:54.361267    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:54.361365 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:54.361340    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:55.155857 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:55.155827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:55.156277 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:55.155874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:55.156277 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.155977    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:55.156277 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.156024    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret podName:048bfc0b-24ff-4316-8586-8a828d189524 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.156011084 +0000 UTC m=+65.171590607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret") pod "global-pull-secret-syncer-t5z88" (UID: "048bfc0b-24ff-4316-8586-8a828d189524") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:55.156277 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.155977    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:55.156277 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.156127    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.156103356 +0000 UTC m=+65.171682892 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:55.256607 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:55.256571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:55.256793 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.256752    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:55.256793 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.256775    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:55.256793 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.256787    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bmr5k for pod openshift-network-diagnostics/network-check-target-2kb8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:55.256946 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:55.256853    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k podName:4fd1744c-8214-49ea-b977-4dfb8f751960 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.256833676 +0000 UTC m=+65.272413198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmr5k" (UniqueName: "kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k") pod "network-check-target-2kb8s" (UID: "4fd1744c-8214-49ea-b977-4dfb8f751960") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:56.361201 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.361114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:56.361649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.361114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:56.361649 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.361232    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kb8s" podUID="4fd1744c-8214-49ea-b977-4dfb8f751960"
Jul 01 18:37:56.361649 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.361323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4pb8p" podUID="03bffaf8-f1aa-48f2-a03f-e2161a1e7e86"
Jul 01 18:37:56.361649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.361119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:56.361649 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.361396    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z88" podUID="048bfc0b-24ff-4316-8586-8a828d189524"
Jul 01 18:37:56.373360 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.373325    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeReady"
Jul 01 18:37:56.373512 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.373447    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 18:37:56.418734 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.418325    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:37:56.420621 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.420597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.423138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.423116    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 18:37:56.423138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.423117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 18:37:56.423318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.423153    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 18:37:56.423318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.423209    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qpglp\""
Jul 01 18:37:56.432047 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.431846    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 18:37:56.434355 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.434332    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s4lpv"]
Jul 01 18:37:56.436335 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.436313    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-24wpt"]
Jul 01 18:37:56.436446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.436355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.438001 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.437979    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.438561 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.438541    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 18:37:56.438561 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.438557    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 18:37:56.438730 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.438713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-555k5\""
Jul 01 18:37:56.443813 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.443644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 18:37:56.445353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.445336    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:37:56.445451 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.445441    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sh8b7\""
Jul 01 18:37:56.446487 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.446430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 18:37:56.447897 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.447816    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 18:37:56.453232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.453212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s4lpv"]
Jul 01 18:37:56.455677 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.455635    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-24wpt"]
Jul 01 18:37:56.565641 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.565798 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.565798 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae58e6-d31b-4218-a13e-826368c57ed1-config-volume\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.565798 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.565798 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aa96c-ede2-4d93-a113-a362c3820669-cert\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2kg\" (UniqueName: \"kubernetes.io/projected/7fae58e6-d31b-4218-a13e-826368c57ed1-kube-api-access-sh2kg\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fae58e6-d31b-4218-a13e-826368c57ed1-tmp-dir\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtql\" (UniqueName: \"kubernetes.io/projected/0c3aa96c-ede2-4d93-a113-a362c3820669-kube-api-access-vhtql\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.565979 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.565954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.566200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.566016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.566200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.566047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.566200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.566081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvsjg\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.666630 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fae58e6-d31b-4218-a13e-826368c57ed1-tmp-dir\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtql\" (UniqueName: \"kubernetes.io/projected/0c3aa96c-ede2-4d93-a113-a362c3820669-kube-api-access-vhtql\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.666790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvsjg\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae58e6-d31b-4218-a13e-826368c57ed1-config-volume\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aa96c-ede2-4d93-a113-a362c3820669-cert\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.666976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.667008    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fae58e6-d31b-4218-a13e-826368c57ed1-tmp-dir\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.667049    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:37:56.667108 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.667066    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:37:56.667539 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.667121    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:57.167100295 +0000 UTC m=+35.182679821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:37:56.667539 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.667139    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.667539 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.667172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2kg\" (UniqueName: \"kubernetes.io/projected/7fae58e6-d31b-4218-a13e-826368c57ed1-kube-api-access-sh2kg\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.667539 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.667329    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:56.667539 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:56.667373    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:57.167359645 +0000 UTC m=+35.182939168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:37:56.667788 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.667673    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae58e6-d31b-4218-a13e-826368c57ed1-config-volume\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.667837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.667782    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.668547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.668517    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.671715 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.671589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.671800 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.671599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.671800 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.671736    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aa96c-ede2-4d93-a113-a362c3820669-cert\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.681574 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.681541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.682819 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.682772    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2kg\" (UniqueName: \"kubernetes.io/projected/7fae58e6-d31b-4218-a13e-826368c57ed1-kube-api-access-sh2kg\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:56.683300 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.683268    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvsjg\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:56.683369 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.683268    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtql\" (UniqueName: \"kubernetes.io/projected/0c3aa96c-ede2-4d93-a113-a362c3820669-kube-api-access-vhtql\") pod \"ingress-canary-24wpt\" (UID: \"0c3aa96c-ede2-4d93-a113-a362c3820669\") " pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:56.753853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:56.753825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-24wpt"
Jul 01 18:37:57.170156 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:57.170123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:57.170346 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:57.170229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:57.170346 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:57.170308    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:57.170461 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:57.170360    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:37:57.170461 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:57.170373    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:37:57.170461 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:57.170394    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:58.170372658 +0000 UTC m=+36.185952196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:37:57.170461 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:57.170415    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:58.1704053 +0000 UTC m=+36.185984827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:37:57.651114 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:57.651090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-24wpt"]
Jul 01 18:37:57.654513 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:37:57.654491    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3aa96c_ede2_4d93_a113_a362c3820669.slice/crio-bfb1949187e2d117c2f1db3891293758379e24f6ca98743bfb43400a24712093 WatchSource:0}: Error finding container bfb1949187e2d117c2f1db3891293758379e24f6ca98743bfb43400a24712093: Status 404 returned error can't find the container with id bfb1949187e2d117c2f1db3891293758379e24f6ca98743bfb43400a24712093
Jul 01 18:37:58.178879 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.178842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:37:58.179014 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.178902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:37:58.179014 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:58.178985    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:37:58.179014 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:58.179005    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:37:58.179014 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:58.179013    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.179203 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:58.179071    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:00.179052104 +0000 UTC m=+38.194631632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:37:58.179203 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:37:58.179090    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:00.179081475 +0000 UTC m=+38.194661030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.365487 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.365461    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:37:58.365634 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.365497    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:37:58.365634 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.365465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:37:58.369219 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369199    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:37:58.369219 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t4jbq\""
Jul 01 18:37:58.369452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qh8tt\""
Jul 01 18:37:58.369824 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:37:58.369824 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:37:58.369824 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.369748    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:37:58.652341 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.652299    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="6a95391aeed4b6aea42ffccba436aa87a544aa88661ecc9cbe620b3cf379c234" exitCode=0
Jul 01 18:37:58.652755 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.652382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"6a95391aeed4b6aea42ffccba436aa87a544aa88661ecc9cbe620b3cf379c234"}
Jul 01 18:37:58.656455 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:58.654243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-24wpt" event={"ID":"0c3aa96c-ede2-4d93-a113-a362c3820669","Type":"ContainerStarted","Data":"bfb1949187e2d117c2f1db3891293758379e24f6ca98743bfb43400a24712093"}
Jul 01 18:37:59.661468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:59.661439    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a63e83e-5bc8-417d-9d12-ccb369d53076" containerID="a684be10b8ac707c8b4bc0d4f6c39f22fe87acb3c5a44f2b5d703de4b673bf1b" exitCode=0
Jul 01 18:37:59.661814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:37:59.661503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerDied","Data":"a684be10b8ac707c8b4bc0d4f6c39f22fe87acb3c5a44f2b5d703de4b673bf1b"}
Jul 01 18:38:00.194677 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.194495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:38:00.194822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.194698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:38:00.194822 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:00.194645    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:00.194822 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:00.194760    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:38:00.194822 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:00.194816    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:00.194958 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:00.194817    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:04.194801834 +0000 UTC m=+42.210381356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:38:00.194958 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:00.194860    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:04.194849848 +0000 UTC m=+42.210429370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:00.666873 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.666837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" event={"ID":"8a63e83e-5bc8-417d-9d12-ccb369d53076","Type":"ContainerStarted","Data":"501aecc220795e790f80566f1d5a513892778cfd34c6ff9447a7773195b921e4"}
Jul 01 18:38:00.668415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.668390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-24wpt" event={"ID":"0c3aa96c-ede2-4d93-a113-a362c3820669","Type":"ContainerStarted","Data":"addbaa7fd1f27704dda63fcc87086c54707e01c077132499d6e527d06608a187"}
Jul 01 18:38:00.701835 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.701795    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2mg9d" podStartSLOduration=9.240413449 podStartE2EDuration="38.701781416s" totalImagesPullingTime="29.461367967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:00.700521475 +0000 UTC m=+38.716101019" watchObservedRunningTime="2026-07-01 18:38:00.701781416 +0000 UTC m=+38.717360963"
Jul 01 18:38:00.721482 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:00.721441    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-24wpt" podStartSLOduration=2.579475624 podStartE2EDuration="4.721427186s" totalImagesPullingTime="2.141951562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:00.721119219 +0000 UTC m=+38.736698767" watchObservedRunningTime="2026-07-01 18:38:00.721427186 +0000 UTC m=+38.737006732"
Jul 01 18:38:03.005656 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.005622    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"]
Jul 01 18:38:03.036874 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.036849    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"]
Jul 01 18:38:03.037010 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.036931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.039369 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.039346    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 18:38:03.041469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.041429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 18:38:03.041469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.041445    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 18:38:03.041603 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.041500    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f8tgn\""
Jul 01 18:38:03.042218 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.042199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 18:38:03.215483 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.215462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ph59\" (UniqueName: \"kubernetes.io/projected/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-kube-api-access-5ph59\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.215591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.215534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.215591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.215557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.316485 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.316425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ph59\" (UniqueName: \"kubernetes.io/projected/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-kube-api-access-5ph59\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.316565 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.316500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.316565 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.316517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.316640 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:03.316608    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:03.316681 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:03.316651    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:03.816638776 +0000 UTC m=+41.832218299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:03.317165 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.317148    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.327766 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.327739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ph59\" (UniqueName: \"kubernetes.io/projected/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-kube-api-access-5ph59\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.820854 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:03.820821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:03.820970 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:03.820937    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:03.821021 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:03.820996    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:04.820978118 +0000 UTC m=+42.836557659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:04.224889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:04.224861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:04.224900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.224999    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.225015    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.225034    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.225047    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:12.225032254 +0000 UTC m=+50.240611775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:04.225235 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.225086    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:12.225068677 +0000 UTC m=+50.240648205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:38:04.830014 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:04.829984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:04.830147 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.830126    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:04.830207 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:04.830187    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:06.830173096 +0000 UTC m=+44.845752623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:06.844496 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:06.844456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:06.844986 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:06.844588    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:06.844986 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:06.844658    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:10.844642575 +0000 UTC m=+48.860222097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:07.100769 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.100699    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pzctv"]
Jul 01 18:38:07.126527 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.126495    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzctv"]
Jul 01 18:38:07.126651 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.126638    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.130300 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.130266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 18:38:07.130429 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.130400    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 18:38:07.130559 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.130542    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 18:38:07.131006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.130988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zc75h\""
Jul 01 18:38:07.131502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.131484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 18:38:07.246994 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.246974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.247097 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.247020    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e916a9c4-a7ad-458b-8250-343d315cb106-data-volume\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.247097 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.247054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e916a9c4-a7ad-458b-8250-343d315cb106-crio-socket\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.247097 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.247084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.247208 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.247101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfcj\" (UniqueName: \"kubernetes.io/projected/e916a9c4-a7ad-458b-8250-343d315cb106-kube-api-access-tzfcj\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347440 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e916a9c4-a7ad-458b-8250-343d315cb106-crio-socket\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347506 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347506 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfcj\" (UniqueName: \"kubernetes.io/projected/e916a9c4-a7ad-458b-8250-343d315cb106-kube-api-access-tzfcj\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347571 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347571 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.347542    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:07.347660 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e916a9c4-a7ad-458b-8250-343d315cb106-data-volume\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347660 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.347599    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:07.847581824 +0000 UTC m=+45.863161350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:07.347660 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347635    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e916a9c4-a7ad-458b-8250-343d315cb106-crio-socket\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.347807 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.347692    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:07.347807 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.347761    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:07.847743494 +0000 UTC m=+45.863323023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:07.347877 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.347824    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e916a9c4-a7ad-458b-8250-343d315cb106-data-volume\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.362428 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.362386    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfcj\" (UniqueName: \"kubernetes.io/projected/e916a9c4-a7ad-458b-8250-343d315cb106-kube-api-access-tzfcj\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.852038 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.852013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.852405 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:07.852073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:07.852405 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.852125    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:07.852405 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.852183    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:07.852405 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.852190    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:08.85217707 +0000 UTC m=+46.867756592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:07.852405 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:07.852220    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:08.852209519 +0000 UTC m=+46.867789041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:08.470178 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:08.470152    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64k9d_0d52c436-737c-49d3-bf32-a3c80dec20e1/dns-node-resolver/0.log"
Jul 01 18:38:08.859957 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:08.859894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:08.860273 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:08.859988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:08.860273 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:08.860050    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:08.860273 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:08.860092    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:08.860273 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:08.860118    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:10.860101795 +0000 UTC m=+48.875681334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:08.860273 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:08.860142    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:10.860129231 +0000 UTC m=+48.875708753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:09.670914 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:09.670880    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mrwgc_ef1f2162-8c0c-42df-87fb-ae2da76b0cd0/node-ca/0.log"
Jul 01 18:38:10.079132 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:10.079075    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-24wpt_0c3aa96c-ede2-4d93-a113-a362c3820669/serve-healthcheck-canary/0.log"
Jul 01 18:38:10.873663 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:10.873632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:10.873663 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:10.873664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:10.873840 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873737    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:10.873840 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:10.873742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:10.873840 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873788    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:14.873775081 +0000 UTC m=+52.889354603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:10.873840 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873839    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:10.873980 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873852    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:10.873980 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873896    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:18.873888477 +0000 UTC m=+56.889467998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:10.873980 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:10.873908    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:14.873901877 +0000 UTC m=+52.889481399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:12.283515 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:12.283485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:12.283527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:12.283625    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:12.283643    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:12.283679    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:12.283703    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.283688133 +0000 UTC m=+66.299267655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:38:12.283870 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:12.283721    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.283709699 +0000 UTC m=+66.299289220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:12.969350 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:12.969322    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"]
Jul 01 18:38:13.002974 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.002953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"]
Jul 01 18:38:13.003109 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.003037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.005402 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.005373    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:13.005527 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.005467    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nzb9h\""
Jul 01 18:38:13.005527 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.005473    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 18:38:13.006225 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.006193    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:13.089532 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.089512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6kt\" (UniqueName: \"kubernetes.io/projected/4d5e170f-11f1-4e31-88d1-a58571d06ec2-kube-api-access-4m6kt\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.089639 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.089570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.190884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.190865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6kt\" (UniqueName: \"kubernetes.io/projected/4d5e170f-11f1-4e31-88d1-a58571d06ec2-kube-api-access-4m6kt\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.190954 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.190916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.191020 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:13.191007    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:13.191072 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:13.191062    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls podName:4d5e170f-11f1-4e31-88d1-a58571d06ec2 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:13.691045296 +0000 UTC m=+51.706624820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zx9x9" (UID: "4d5e170f-11f1-4e31-88d1-a58571d06ec2") : secret "samples-operator-tls" not found
Jul 01 18:38:13.199717 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.199691    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6kt\" (UniqueName: \"kubernetes.io/projected/4d5e170f-11f1-4e31-88d1-a58571d06ec2-kube-api-access-4m6kt\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.695354 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:13.695326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:13.695708 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:13.695509    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:13.695708 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:13.695574    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls podName:4d5e170f-11f1-4e31-88d1-a58571d06ec2 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:14.695556908 +0000 UTC m=+52.711136436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zx9x9" (UID: "4d5e170f-11f1-4e31-88d1-a58571d06ec2") : secret "samples-operator-tls" not found
Jul 01 18:38:14.701789 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:14.701759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:14.702155 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:14.701903    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:14.702155 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:14.701958    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls podName:4d5e170f-11f1-4e31-88d1-a58571d06ec2 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:16.701942591 +0000 UTC m=+54.717522113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zx9x9" (UID: "4d5e170f-11f1-4e31-88d1-a58571d06ec2") : secret "samples-operator-tls" not found
Jul 01 18:38:14.904253 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:14.904224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:14.904397 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:14.904347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:14.904397 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:14.904385    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:14.904469 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:14.904439    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:22.904426046 +0000 UTC m=+60.920005571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:14.904847 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:14.904829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e916a9c4-a7ad-458b-8250-343d315cb106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:16.718016 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:16.717979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:16.718503 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:16.718094    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:16.718503 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:16.718150    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls podName:4d5e170f-11f1-4e31-88d1-a58571d06ec2 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:20.718136479 +0000 UTC m=+58.733716001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zx9x9" (UID: "4d5e170f-11f1-4e31-88d1-a58571d06ec2") : secret "samples-operator-tls" not found
Jul 01 18:38:18.935596 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:18.935563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:18.935966 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:18.935680    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:18.935966 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:18.935730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls podName:4e91f2e0-0cf9-44d1-b2cb-f915acb5a399 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:34.935716954 +0000 UTC m=+72.951296476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-9vsdc" (UID: "4e91f2e0-0cf9-44d1-b2cb-f915acb5a399") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:20.749760 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:20.749733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:20.750132 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:20.749903    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:20.750132 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:20.749962    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls podName:4d5e170f-11f1-4e31-88d1-a58571d06ec2 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.749943143 +0000 UTC m=+66.765522667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zx9x9" (UID: "4d5e170f-11f1-4e31-88d1-a58571d06ec2") : secret "samples-operator-tls" not found
Jul 01 18:38:22.649322 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:22.649276    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxvb"
Jul 01 18:38:22.965475 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:22.965403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:22.965677 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:22.965534    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:22.965677 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:22.965597    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls podName:e916a9c4-a7ad-458b-8250-343d315cb106 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:38.965581098 +0000 UTC m=+76.981160625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzctv" (UID: "e916a9c4-a7ad-458b-8250-343d315cb106") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:27.196197 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.196161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:38:27.196715 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.196303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:38:27.199126 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.199105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:38:27.199593 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.199578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:38:27.207339 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:27.207319    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 18:38:27.207407 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:27.207394    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs podName:03bffaf8-f1aa-48f2-a03f-e2161a1e7e86 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:31.207372326 +0000 UTC m=+129.222951866 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs") pod "network-metrics-daemon-4pb8p" (UID: "03bffaf8-f1aa-48f2-a03f-e2161a1e7e86") : secret "metrics-daemon-secret" not found
Jul 01 18:38:27.209672 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.209651    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048bfc0b-24ff-4316-8586-8a828d189524-original-pull-secret\") pod \"global-pull-secret-syncer-t5z88\" (UID: \"048bfc0b-24ff-4316-8586-8a828d189524\") " pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:38:27.297114 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.297083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:38:27.300007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.299990    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:38:27.310243 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.310222    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:38:27.320263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.320234    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmr5k\" (UniqueName: \"kubernetes.io/projected/4fd1744c-8214-49ea-b977-4dfb8f751960-kube-api-access-bmr5k\") pod \"network-check-target-2kb8s\" (UID: \"4fd1744c-8214-49ea-b977-4dfb8f751960\") " pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:38:27.477494 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.477429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z88"
Jul 01 18:38:27.493041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.493019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qh8tt\""
Jul 01 18:38:27.500934 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.500906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:38:27.614658 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.614633    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5z88"]
Jul 01 18:38:27.618106 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:27.618083    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048bfc0b_24ff_4316_8586_8a828d189524.slice/crio-75d33a2b44040df4c988c2ac039e3992ee9a5b2a7002b1648bab5f1975e4fa20 WatchSource:0}: Error finding container 75d33a2b44040df4c988c2ac039e3992ee9a5b2a7002b1648bab5f1975e4fa20: Status 404 returned error can't find the container with id 75d33a2b44040df4c988c2ac039e3992ee9a5b2a7002b1648bab5f1975e4fa20
Jul 01 18:38:27.664179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.664159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kb8s"]
Jul 01 18:38:27.666419 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:27.666396    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1744c_8214_49ea_b977_4dfb8f751960.slice/crio-8849ddf8dfac4aa0d803c974a259c233355431fcc2f428e200d5cf71fd3f8eda WatchSource:0}: Error finding container 8849ddf8dfac4aa0d803c974a259c233355431fcc2f428e200d5cf71fd3f8eda: Status 404 returned error can't find the container with id 8849ddf8dfac4aa0d803c974a259c233355431fcc2f428e200d5cf71fd3f8eda
Jul 01 18:38:27.728974 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.728921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5z88" event={"ID":"048bfc0b-24ff-4316-8586-8a828d189524","Type":"ContainerStarted","Data":"75d33a2b44040df4c988c2ac039e3992ee9a5b2a7002b1648bab5f1975e4fa20"}
Jul 01 18:38:27.729860 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:27.729841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kb8s" event={"ID":"4fd1744c-8214-49ea-b977-4dfb8f751960","Type":"ContainerStarted","Data":"8849ddf8dfac4aa0d803c974a259c233355431fcc2f428e200d5cf71fd3f8eda"}
Jul 01 18:38:28.303825 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.303784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.303844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:28.303943    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:28.303954    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:28.303974    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b69d6978-m4vw7: secret "image-registry-tls" not found
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:28.304005    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls podName:7fae58e6-d31b-4218-a13e-826368c57ed1 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:00.30398941 +0000 UTC m=+98.319568935 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls") pod "dns-default-s4lpv" (UID: "7fae58e6-d31b-4218-a13e-826368c57ed1") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:28.304228 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:28.304029    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls podName:2ea0e6b4-3c99-4716-b31b-746a9eb671c4 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:00.3040107 +0000 UTC m=+98.319590248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls") pod "image-registry-58b69d6978-m4vw7" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4") : secret "image-registry-tls" not found
Jul 01 18:38:28.807153 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.807120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:28.810072 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.810049    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d5e170f-11f1-4e31-88d1-a58571d06ec2-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zx9x9\" (UID: \"4d5e170f-11f1-4e31-88d1-a58571d06ec2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:28.915065 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.915037    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nzb9h\""
Jul 01 18:38:28.922350 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:28.922329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"
Jul 01 18:38:29.101033 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:29.101002    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9"]
Jul 01 18:38:29.738638 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:29.738531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9" event={"ID":"4d5e170f-11f1-4e31-88d1-a58571d06ec2","Type":"ContainerStarted","Data":"5692af0b3765fe65f36c84f71aa8aac59b1bc8109d7192224ac2aaacf269ba1d"}
Jul 01 18:38:32.749651 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.749616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kb8s" event={"ID":"4fd1744c-8214-49ea-b977-4dfb8f751960","Type":"ContainerStarted","Data":"acf94c4a37f810de9888231f84b182fc4101ca72823da449fc8121f25b0ca9ac"}
Jul 01 18:38:32.750087 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.749708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:38:32.751385 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.751358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9" event={"ID":"4d5e170f-11f1-4e31-88d1-a58571d06ec2","Type":"ContainerStarted","Data":"a7b38a940a9dd9a868065f14858d543fb5775666e129913314dbf98eb4c8a6c2"}
Jul 01 18:38:32.751524 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.751395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9" event={"ID":"4d5e170f-11f1-4e31-88d1-a58571d06ec2","Type":"ContainerStarted","Data":"bf43a6e442bdcd5f16af882cd42afd9a8fb70b65a702e7bf516ce27119f983f6"}
Jul 01 18:38:32.752757 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.752737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5z88" event={"ID":"048bfc0b-24ff-4316-8586-8a828d189524","Type":"ContainerStarted","Data":"ca95a8ecaf6df367a05622a5112e96eaaaf8e837cb61e04c618983b00fbc3013"}
Jul 01 18:38:32.776042 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.776000    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2kb8s" podStartSLOduration=65.897617537 podStartE2EDuration="1m10.775988018s" totalImagesPullingTime="4.878370481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:32.775568493 +0000 UTC m=+70.791148062" watchObservedRunningTime="2026-07-01 18:38:32.775988018 +0000 UTC m=+70.791567562"
Jul 01 18:38:32.803598 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:32.803515    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zx9x9" podStartSLOduration=17.401335504 podStartE2EDuration="20.803503808s" totalImagesPullingTime="3.402168304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:32.803474939 +0000 UTC m=+70.819054484" watchObservedRunningTime="2026-07-01 18:38:32.803503808 +0000 UTC m=+70.819083351"
Jul 01 18:38:34.954911 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:34.954868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:34.957422 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:34.957396    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e91f2e0-0cf9-44d1-b2cb-f915acb5a399-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-9vsdc\" (UID: \"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:35.148200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:35.148172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f8tgn\""
Jul 01 18:38:35.156166 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:35.156146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"
Jul 01 18:38:35.286977 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:35.286932    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t5z88" podStartSLOduration=68.337120292 podStartE2EDuration="1m13.286910453s" totalImagesPullingTime="4.949790161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:32.822333433 +0000 UTC m=+70.837912976" watchObservedRunningTime="2026-07-01 18:38:35.286910453 +0000 UTC m=+73.302489991"
Jul 01 18:38:35.287902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:35.287876    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc"]
Jul 01 18:38:35.292297 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:35.292248    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e91f2e0_0cf9_44d1_b2cb_f915acb5a399.slice/crio-5b960ba4573af77e215d70fab3ac7dcc9d357fdac101b6011277bac91f920010 WatchSource:0}: Error finding container 5b960ba4573af77e215d70fab3ac7dcc9d357fdac101b6011277bac91f920010: Status 404 returned error can't find the container with id 5b960ba4573af77e215d70fab3ac7dcc9d357fdac101b6011277bac91f920010
Jul 01 18:38:35.766142 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:35.766103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc" event={"ID":"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399","Type":"ContainerStarted","Data":"5b960ba4573af77e215d70fab3ac7dcc9d357fdac101b6011277bac91f920010"}
Jul 01 18:38:37.774159 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:37.774127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc" event={"ID":"4e91f2e0-0cf9-44d1-b2cb-f915acb5a399","Type":"ContainerStarted","Data":"73633ee9b049e340fe4733b84788dd5f5b9022b4e422ae28c05baab10a205d42"}
Jul 01 18:38:37.812700 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:37.812653    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-9vsdc" podStartSLOduration=33.9667365 podStartE2EDuration="35.812638603s" totalImagesPullingTime="1.845902103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:37.811263107 +0000 UTC m=+75.826842654" watchObservedRunningTime="2026-07-01 18:38:37.812638603 +0000 UTC m=+75.828218192"
Jul 01 18:38:38.984269 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:38.984227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:38.986701 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:38.986674    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e916a9c4-a7ad-458b-8250-343d315cb106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzctv\" (UID: \"e916a9c4-a7ad-458b-8250-343d315cb106\") " pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:39.238313 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:39.238228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zc75h\""
Jul 01 18:38:39.246351 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:39.246329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzctv"
Jul 01 18:38:39.389350 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:39.389179    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzctv"]
Jul 01 18:38:39.392043 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:39.392021    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode916a9c4_a7ad_458b_8250_343d315cb106.slice/crio-d4f43411fb27f1d29645bd18f35f4b0c181a2919553da26c70663e55d705a738 WatchSource:0}: Error finding container d4f43411fb27f1d29645bd18f35f4b0c181a2919553da26c70663e55d705a738: Status 404 returned error can't find the container with id d4f43411fb27f1d29645bd18f35f4b0c181a2919553da26c70663e55d705a738
Jul 01 18:38:39.783550 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:39.783515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzctv" event={"ID":"e916a9c4-a7ad-458b-8250-343d315cb106","Type":"ContainerStarted","Data":"d4f43411fb27f1d29645bd18f35f4b0c181a2919553da26c70663e55d705a738"}
Jul 01 18:38:43.574409 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.574380    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"]
Jul 01 18:38:43.580040 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.580014    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.586208 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.586184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 18:38:43.587977 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.587956    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 18:38:43.588064 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.587958    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 18:38:43.588545 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.588525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 18:38:43.636436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.636404    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"]
Jul 01 18:38:43.639173 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.639154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.642477 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.642460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cvtnq\""
Jul 01 18:38:43.642731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.642708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 18:38:43.643152 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.643137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 18:38:43.661072 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.661051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"]
Jul 01 18:38:43.671649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.671625    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"]
Jul 01 18:38:43.722726 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.722704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73a72b23-be7c-4fac-8e30-5d5862adde93-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.722807 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.722759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-tmp\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.722883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.722822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.722916 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.722883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73a72b23-be7c-4fac-8e30-5d5862adde93-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.722997 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.722979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxvl\" (UniqueName: \"kubernetes.io/projected/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-kube-api-access-wgxvl\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.778584 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.778560    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"]
Jul 01 18:38:43.781443 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.781430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:43.790720 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.790704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 01 18:38:43.791031 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.791018    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xw8nh\""
Jul 01 18:38:43.797941 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.797916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzctv" event={"ID":"e916a9c4-a7ad-458b-8250-343d315cb106","Type":"ContainerStarted","Data":"a8fe7e627f71f459a1b5e25bc94f7eaed2149e425e8772ac37217039ed4e49ac"}
Jul 01 18:38:43.798011 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.797962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzctv" event={"ID":"e916a9c4-a7ad-458b-8250-343d315cb106","Type":"ContainerStarted","Data":"70f646a42ca4031975222a1a436bb1cf10eb09d291a0636036880ad0dd43981a"}
Jul 01 18:38:43.811110 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.811090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"]
Jul 01 18:38:43.824071 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.824168 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73a72b23-be7c-4fac-8e30-5d5862adde93-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.824168 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxvl\" (UniqueName: \"kubernetes.io/projected/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-kube-api-access-wgxvl\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.824168 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73a72b23-be7c-4fac-8e30-5d5862adde93-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.824359 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-tmp\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.824600 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-tmp\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.824816 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.824796    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73a72b23-be7c-4fac-8e30-5d5862adde93-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.826378 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.826355    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.826584 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.826546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73a72b23-be7c-4fac-8e30-5d5862adde93-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-hswzv\" (UID: \"73a72b23-be7c-4fac-8e30-5d5862adde93\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:43.850435 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.850414    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxvl\" (UniqueName: \"kubernetes.io/projected/a829f8a9-0968-4110-9c1c-9d5e2c340fb2-kube-api-access-wgxvl\") pod \"klusterlet-addon-workmgr-77647987b9-hzjdk\" (UID: \"a829f8a9-0968-4110-9c1c-9d5e2c340fb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.890297 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.890263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:43.925409 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.925386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/891e41bf-d175-4dcd-9c68-8bbef7bfa9c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-52lg5\" (UID: \"891e41bf-d175-4dcd-9c68-8bbef7bfa9c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:43.947576 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:43.947557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"
Jul 01 18:38:44.026777 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.026742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/891e41bf-d175-4dcd-9c68-8bbef7bfa9c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-52lg5\" (UID: \"891e41bf-d175-4dcd-9c68-8bbef7bfa9c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:44.029825 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.029800    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/891e41bf-d175-4dcd-9c68-8bbef7bfa9c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-52lg5\" (UID: \"891e41bf-d175-4dcd-9c68-8bbef7bfa9c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:44.036836 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.036810    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"]
Jul 01 18:38:44.039471 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:44.039446    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda829f8a9_0968_4110_9c1c_9d5e2c340fb2.slice/crio-bc835c70db8aba822a14401b4b5de158cf0c27af7f240cb7d1c01108d2742931 WatchSource:0}: Error finding container bc835c70db8aba822a14401b4b5de158cf0c27af7f240cb7d1c01108d2742931: Status 404 returned error can't find the container with id bc835c70db8aba822a14401b4b5de158cf0c27af7f240cb7d1c01108d2742931
Jul 01 18:38:44.089056 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.089000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:44.092694 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.092662    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv"]
Jul 01 18:38:44.097650 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:44.097628    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a72b23_be7c_4fac_8e30_5d5862adde93.slice/crio-5cdf5329a0f70369295a037d3bebfd0fddd228ff5afbc0c4e3b9aef850a567b6 WatchSource:0}: Error finding container 5cdf5329a0f70369295a037d3bebfd0fddd228ff5afbc0c4e3b9aef850a567b6: Status 404 returned error can't find the container with id 5cdf5329a0f70369295a037d3bebfd0fddd228ff5afbc0c4e3b9aef850a567b6
Jul 01 18:38:44.266205 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.266172    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"]
Jul 01 18:38:44.269999 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:44.269973    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891e41bf_d175_4dcd_9c68_8bbef7bfa9c8.slice/crio-179e4bff091906a8728d0974d51a8c45f548c2e705bb26fa9105214a45a5ea6c WatchSource:0}: Error finding container 179e4bff091906a8728d0974d51a8c45f548c2e705bb26fa9105214a45a5ea6c: Status 404 returned error can't find the container with id 179e4bff091906a8728d0974d51a8c45f548c2e705bb26fa9105214a45a5ea6c
Jul 01 18:38:44.805275 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.805188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk" event={"ID":"a829f8a9-0968-4110-9c1c-9d5e2c340fb2","Type":"ContainerStarted","Data":"bc835c70db8aba822a14401b4b5de158cf0c27af7f240cb7d1c01108d2742931"}
Jul 01 18:38:44.807474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.807442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5" event={"ID":"891e41bf-d175-4dcd-9c68-8bbef7bfa9c8","Type":"ContainerStarted","Data":"179e4bff091906a8728d0974d51a8c45f548c2e705bb26fa9105214a45a5ea6c"}
Jul 01 18:38:44.811513 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:44.811487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv" event={"ID":"73a72b23-be7c-4fac-8e30-5d5862adde93","Type":"ContainerStarted","Data":"5cdf5329a0f70369295a037d3bebfd0fddd228ff5afbc0c4e3b9aef850a567b6"}
Jul 01 18:38:47.246057 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.245979    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:38:47.249495 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.249476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.253764 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.253742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 18:38:47.254823 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.254800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 18:38:47.255165 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.255138    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 18:38:47.255261 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.255179    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xvvhz\""
Jul 01 18:38:47.255424 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.255405    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 18:38:47.255823 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.255806    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 18:38:47.256119 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.256098    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 18:38:47.257963 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.257944    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 18:38:47.265975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.265951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:38:47.354835 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.354835 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.354976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.354976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.354976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn595\" (UniqueName: \"kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.354976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.354973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455780 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455825 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455866 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn595\" (UniqueName: \"kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.455917 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.455862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.456346 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.456322    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.456465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.456425    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.456549 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.456533    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.458384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.458358    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.458467 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.458408    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.477218 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.477195    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn595\" (UniqueName: \"kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595\") pod \"console-6d95b95bb-ghv6d\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") " pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.560995 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.560942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:47.716165 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.716145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:38:47.718177 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:47.718156    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccf9274_6105_4faf_8e15_258cec365b67.slice/crio-d9eae5053e0183f439914666fcda45950a604df7074932ce43de093aed486d14 WatchSource:0}: Error finding container d9eae5053e0183f439914666fcda45950a604df7074932ce43de093aed486d14: Status 404 returned error can't find the container with id d9eae5053e0183f439914666fcda45950a604df7074932ce43de093aed486d14
Jul 01 18:38:47.823599 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.823536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv" event={"ID":"73a72b23-be7c-4fac-8e30-5d5862adde93","Type":"ContainerStarted","Data":"a8b3f18bc2eb4f6139300268b73ac0e872f5850153df3097864876e8ced212e6"}
Jul 01 18:38:47.824817 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.824791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d95b95bb-ghv6d" event={"ID":"6ccf9274-6105-4faf-8e15-258cec365b67","Type":"ContainerStarted","Data":"d9eae5053e0183f439914666fcda45950a604df7074932ce43de093aed486d14"}
Jul 01 18:38:47.826799 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.826764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzctv" event={"ID":"e916a9c4-a7ad-458b-8250-343d315cb106","Type":"ContainerStarted","Data":"6122218e9099d1201ebc7848ac825952a806330db4ab079ba98ed0df0b232d3c"}
Jul 01 18:38:47.828023 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.827999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5" event={"ID":"891e41bf-d175-4dcd-9c68-8bbef7bfa9c8","Type":"ContainerStarted","Data":"30da60d4d1f6664a631aae406e1d148d6c530dd3ed5aae1496117403745125e2"}
Jul 01 18:38:47.828211 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.828195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:47.832610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.832594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5"
Jul 01 18:38:47.846483 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.846447    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-hswzv" podStartSLOduration=2.028207176 podStartE2EDuration="4.846434207s" totalImagesPullingTime="2.818227031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:47.845487151 +0000 UTC m=+85.861066708" watchObservedRunningTime="2026-07-01 18:38:47.846434207 +0000 UTC m=+85.862013729"
Jul 01 18:38:47.873111 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:47.873068    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pzctv" podStartSLOduration=33.443614471 podStartE2EDuration="40.873051267s" totalImagesPullingTime="7.429436796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:38:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:47.872696813 +0000 UTC m=+85.888276359" watchObservedRunningTime="2026-07-01 18:38:47.873051267 +0000 UTC m=+85.888630812"
Jul 01 18:38:48.987187 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.987138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-52lg5" podStartSLOduration=3.341578999 podStartE2EDuration="5.987118054s" totalImagesPullingTime="2.645539055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:47.898480262 +0000 UTC m=+85.914059805" watchObservedRunningTime="2026-07-01 18:38:48.987118054 +0000 UTC m=+87.002697592"
Jul 01 18:38:48.987592 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.987297    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"]
Jul 01 18:38:48.990681 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.990662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:48.995360 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.995336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 18:38:48.996440 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.996414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h2r55\""
Jul 01 18:38:48.996558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.996437    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 18:38:48.996558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:48.996453    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 18:38:49.019033 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.019008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"]
Jul 01 18:38:49.068649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.068620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.068759 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.068652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.068759 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.068721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.068865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.068803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtsq2\" (UniqueName: \"kubernetes.io/projected/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-kube-api-access-mtsq2\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.169858 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.169832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsq2\" (UniqueName: \"kubernetes.io/projected/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-kube-api-access-mtsq2\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.170007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.169910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.170007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.169953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.170007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.169992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.170685 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.170662    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.172677 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.172647    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.172763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.172728    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.185326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.185279    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtsq2\" (UniqueName: \"kubernetes.io/projected/4e74f6c8-ceae-4652-8bc8-9e84089d17ed-kube-api-access-mtsq2\") pod \"prometheus-operator-7b7d896497-v5ccj\" (UID: \"4e74f6c8-ceae-4652-8bc8-9e84089d17ed\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.301945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.301876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"
Jul 01 18:38:49.464976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:49.464943    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-v5ccj"]
Jul 01 18:38:50.491211 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:50.491171    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e74f6c8_ceae_4652_8bc8_9e84089d17ed.slice/crio-b8b8c06b827a9d91a0a8bd1f84e7b27686ddbcbbf4d308367b163edacf11ebd4 WatchSource:0}: Error finding container b8b8c06b827a9d91a0a8bd1f84e7b27686ddbcbbf4d308367b163edacf11ebd4: Status 404 returned error can't find the container with id b8b8c06b827a9d91a0a8bd1f84e7b27686ddbcbbf4d308367b163edacf11ebd4
Jul 01 18:38:50.840124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:50.840037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj" event={"ID":"4e74f6c8-ceae-4652-8bc8-9e84089d17ed","Type":"ContainerStarted","Data":"b8b8c06b827a9d91a0a8bd1f84e7b27686ddbcbbf4d308367b163edacf11ebd4"}
Jul 01 18:38:52.847864 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.847800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk" event={"ID":"a829f8a9-0968-4110-9c1c-9d5e2c340fb2","Type":"ContainerStarted","Data":"0029aa36daae0282bd7af46a334f3dc417a3374ec98957e033ed607f25e68f78"}
Jul 01 18:38:52.848243 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.848231    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:52.849512 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.849481    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk" podUID="a829f8a9-0968-4110-9c1c-9d5e2c340fb2" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.12:8000/readyz\": dial tcp 10.133.0.12:8000: connect: connection refused"
Jul 01 18:38:52.849818 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.849794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d95b95bb-ghv6d" event={"ID":"6ccf9274-6105-4faf-8e15-258cec365b67","Type":"ContainerStarted","Data":"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"}
Jul 01 18:38:52.852036 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.852010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj" event={"ID":"4e74f6c8-ceae-4652-8bc8-9e84089d17ed","Type":"ContainerStarted","Data":"f7a456e94667d05910dcd92f2942bbb018d81f5dc247ca64c92c66d95a02b9af"}
Jul 01 18:38:52.886341 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.886005    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk" podStartSLOduration=1.166186428 podStartE2EDuration="9.885986094s" totalImagesPullingTime="8.719799666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:52.883760091 +0000 UTC m=+90.899339637" watchObservedRunningTime="2026-07-01 18:38:52.885986094 +0000 UTC m=+90.901565630"
Jul 01 18:38:52.914091 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:52.914043    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d95b95bb-ghv6d" podStartSLOduration=1.260899205 podStartE2EDuration="5.91402284s" totalImagesPullingTime="4.653123635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:52.914002657 +0000 UTC m=+90.929582203" watchObservedRunningTime="2026-07-01 18:38:52.91402284 +0000 UTC m=+90.929602385"
Jul 01 18:38:53.856864 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:53.856820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj" event={"ID":"4e74f6c8-ceae-4652-8bc8-9e84089d17ed","Type":"ContainerStarted","Data":"16883f894db8d97077f298e57c982ec293cf36aa59b214a23aaf27eb71646c5d"}
Jul 01 18:38:53.858007 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:53.857983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77647987b9-hzjdk"
Jul 01 18:38:53.879131 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:53.879085    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-v5ccj" podStartSLOduration=3.60756461 podStartE2EDuration="5.879069422s" totalImagesPullingTime="2.271504812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:53.878526464 +0000 UTC m=+91.894106029" watchObservedRunningTime="2026-07-01 18:38:53.879069422 +0000 UTC m=+91.894648965"
Jul 01 18:38:55.628376 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.628343    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8cb98"]
Jul 01 18:38:55.631564 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.631550    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"]
Jul 01 18:38:55.631683 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.631666    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.634427 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.634409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.634775 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.634758    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 18:38:55.634848 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.634831    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2g74v\""
Jul 01 18:38:55.635137 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.635121    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 18:38:55.635349 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.635337    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 18:38:55.636448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.636434    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cl5br\""
Jul 01 18:38:55.636889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.636869    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 18:38:55.637358 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.637344    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 18:38:55.654359 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.654342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8cb98"]
Jul 01 18:38:55.655312 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.655278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"]
Jul 01 18:38:55.663991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.663972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bpcvc"]
Jul 01 18:38:55.667469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.667453    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.670158 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.670143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 18:38:55.670724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.670709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 18:38:55.670800 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.670737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gtflm\""
Jul 01 18:38:55.671852 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.671834    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 18:38:55.724725 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.724822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq2f\" (UniqueName: \"kubernetes.io/projected/7f4b3d00-e5c7-4864-be13-9e696c4952aa-kube-api-access-gpq2f\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.724822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.724822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.724973 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.725027 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.724971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.725080 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.725054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab6700df-741e-4531-855d-d09717ed35d8-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.725118 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.725104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.725160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.725123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4b3d00-e5c7-4864-be13-9e696c4952aa-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.725223 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.725159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrlb\" (UniqueName: \"kubernetes.io/projected/ab6700df-741e-4531-855d-d09717ed35d8-kube-api-access-jtrlb\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.825976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.826105 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq2f\" (UniqueName: \"kubernetes.io/projected/7f4b3d00-e5c7-4864-be13-9e696c4952aa-kube-api-access-gpq2f\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.826105 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826105 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-wtmp\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-metrics-client-ca\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-textfile\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.826400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-sys\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826555 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab6700df-741e-4531-855d-d09717ed35d8-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826616 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgrz\" (UniqueName: \"kubernetes.io/projected/2a7be955-7677-4339-8d73-7ced4799a754-kube-api-access-gvgrz\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826676 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826676 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4b3d00-e5c7-4864-be13-9e696c4952aa-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.826778 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826778 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrlb\" (UniqueName: \"kubernetes.io/projected/ab6700df-741e-4531-855d-d09717ed35d8-kube-api-access-jtrlb\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826778 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-root\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.826975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826923    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.826975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.826937    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ab6700df-741e-4531-855d-d09717ed35d8-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.827182 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.827036    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.827772 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.827749    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4b3d00-e5c7-4864-be13-9e696c4952aa-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.828739 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.828707    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.828812 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.828714    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.828963 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.828940    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f4b3d00-e5c7-4864-be13-9e696c4952aa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.829018 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.829004    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6700df-741e-4531-855d-d09717ed35d8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.838460 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.838440    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq2f\" (UniqueName: \"kubernetes.io/projected/7f4b3d00-e5c7-4864-be13-9e696c4952aa-kube-api-access-gpq2f\") pod \"openshift-state-metrics-57f99c8b49-zhm97\" (UID: \"7f4b3d00-e5c7-4864-be13-9e696c4952aa\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.838926 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.838907    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrlb\" (UniqueName: \"kubernetes.io/projected/ab6700df-741e-4531-855d-d09717ed35d8-kube-api-access-jtrlb\") pod \"kube-state-metrics-845b954455-8cb98\" (UID: \"ab6700df-741e-4531-855d-d09717ed35d8\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.928016 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.927967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-wtmp\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928016 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.927990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-metrics-client-ca\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928016 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-textfile\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-sys\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgrz\" (UniqueName: \"kubernetes.io/projected/2a7be955-7677-4339-8d73-7ced4799a754-kube-api-access-gvgrz\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928119    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-sys\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928169    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-wtmp\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-root\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928331    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a7be955-7677-4339-8d73-7ced4799a754-root\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928347    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-textfile\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:55.928351    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928468 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:38:55.928425    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls podName:2a7be955-7677-4339-8d73-7ced4799a754 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:56.428409888 +0000 UTC m=+94.443989410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls") pod "node-exporter-bpcvc" (UID: "2a7be955-7677-4339-8d73-7ced4799a754") : secret "node-exporter-tls" not found
Jul 01 18:38:55.928655 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928506    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-metrics-client-ca\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.928822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.928804    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.930001 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.929985    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:55.941015 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.940998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98"
Jul 01 18:38:55.945774 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.945756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"
Jul 01 18:38:55.946423 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:55.946406    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgrz\" (UniqueName: \"kubernetes.io/projected/2a7be955-7677-4339-8d73-7ced4799a754-kube-api-access-gvgrz\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:56.090448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.090303    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97"]
Jul 01 18:38:56.093002 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:56.092976    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4b3d00_e5c7_4864_be13_9e696c4952aa.slice/crio-faa9b6f3d7d84374e4efc6543d68ddb8214f28925524ab0b2675e5493b7693ea WatchSource:0}: Error finding container faa9b6f3d7d84374e4efc6543d68ddb8214f28925524ab0b2675e5493b7693ea: Status 404 returned error can't find the container with id faa9b6f3d7d84374e4efc6543d68ddb8214f28925524ab0b2675e5493b7693ea
Jul 01 18:38:56.109940 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.109875    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8cb98"]
Jul 01 18:38:56.112611 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:56.112592    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6700df_741e_4531_855d_d09717ed35d8.slice/crio-6616c0f9b94484ac0af26ef982df3ef6151ccc5538313ab3503703a8fe7a9bdc WatchSource:0}: Error finding container 6616c0f9b94484ac0af26ef982df3ef6151ccc5538313ab3503703a8fe7a9bdc: Status 404 returned error can't find the container with id 6616c0f9b94484ac0af26ef982df3ef6151ccc5538313ab3503703a8fe7a9bdc
Jul 01 18:38:56.431398 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.431363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:56.433425 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.433408    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a7be955-7677-4339-8d73-7ced4799a754-node-exporter-tls\") pod \"node-exporter-bpcvc\" (UID: \"2a7be955-7677-4339-8d73-7ced4799a754\") " pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:56.576417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.576345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpcvc"
Jul 01 18:38:56.587221 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:56.587190    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7be955_7677_4339_8d73_7ced4799a754.slice/crio-bc3c5d5671bbacb5b2d417ea03c3457af4aeff770f3e234dd0ea8302a0e26c7c WatchSource:0}: Error finding container bc3c5d5671bbacb5b2d417ea03c3457af4aeff770f3e234dd0ea8302a0e26c7c: Status 404 returned error can't find the container with id bc3c5d5671bbacb5b2d417ea03c3457af4aeff770f3e234dd0ea8302a0e26c7c
Jul 01 18:38:56.740555 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.740140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:38:56.744589 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.744569    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.747473 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.747436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 18:38:56.747549 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.747514    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lvkv5\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.747740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.747851    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.747453    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.748104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.748107    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.748343    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.748349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 18:38:56.748557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.748353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 18:38:56.767743 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.765998    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:38:56.836127 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836127 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.836624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.836597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wtb\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.871014 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.870980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97" event={"ID":"7f4b3d00-e5c7-4864-be13-9e696c4952aa","Type":"ContainerStarted","Data":"3537e4e439f9959f6d65cdb07307307eba826570fa48fa1bb636ca527301df2d"}
Jul 01 18:38:56.871160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.871051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97" event={"ID":"7f4b3d00-e5c7-4864-be13-9e696c4952aa","Type":"ContainerStarted","Data":"d1d97d0a4b35742e6004f4daa953c18bab2132df94aadad40b21296ea11f66e6"}
Jul 01 18:38:56.871160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.871081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97" event={"ID":"7f4b3d00-e5c7-4864-be13-9e696c4952aa","Type":"ContainerStarted","Data":"faa9b6f3d7d84374e4efc6543d68ddb8214f28925524ab0b2675e5493b7693ea"}
Jul 01 18:38:56.872676 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.872647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpcvc" event={"ID":"2a7be955-7677-4339-8d73-7ced4799a754","Type":"ContainerStarted","Data":"bc3c5d5671bbacb5b2d417ea03c3457af4aeff770f3e234dd0ea8302a0e26c7c"}
Jul 01 18:38:56.874042 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.874019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98" event={"ID":"ab6700df-741e-4531-855d-d09717ed35d8","Type":"ContainerStarted","Data":"6616c0f9b94484ac0af26ef982df3ef6151ccc5538313ab3503703a8fe7a9bdc"}
Jul 01 18:38:56.937274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937397 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937397 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937397 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wtb\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.937842 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.937607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.939094 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.938985    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.939094 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.939014    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.939349 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.939280    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.940513 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.940491    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.940513 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.940509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.940811 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.940770    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.941224 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.941186    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.941900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.941877    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.942685 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.942667    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.942915 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.942869    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.943065 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.943041    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.943642 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.943622    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:56.949720 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:56.949702    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wtb\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb\") pod \"alertmanager-main-0\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:57.057103 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.057073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:38:57.561635 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.561336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:57.562920 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.562130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:57.570551 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.570523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:57.651750 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.651680    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:38:57.659066 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:38:57.659034    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10a91c8_fadb_4427_a8a2_67bc1d83da04.slice/crio-36c2538943ba6567edf3269d7690e13e796c1efba6f9bfe3bb79b88ee0600796 WatchSource:0}: Error finding container 36c2538943ba6567edf3269d7690e13e796c1efba6f9bfe3bb79b88ee0600796: Status 404 returned error can't find the container with id 36c2538943ba6567edf3269d7690e13e796c1efba6f9bfe3bb79b88ee0600796
Jul 01 18:38:57.880946 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.880913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97" event={"ID":"7f4b3d00-e5c7-4864-be13-9e696c4952aa","Type":"ContainerStarted","Data":"1a82f7b9e475dea75bbc0f6d91747b2d7d6d72dd692cac2d04450be30053bb5f"}
Jul 01 18:38:57.883496 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.883420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98" event={"ID":"ab6700df-741e-4531-855d-d09717ed35d8","Type":"ContainerStarted","Data":"c09d7a97aa6de37d02908f1905f1916c1299edde55c066f105abe86fd4e0f1c6"}
Jul 01 18:38:57.883496 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.883476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98" event={"ID":"ab6700df-741e-4531-855d-d09717ed35d8","Type":"ContainerStarted","Data":"1e3bdb18006471ea46dc4212187b48beb63d9d12862ebea54030a23526aa721f"}
Jul 01 18:38:57.883668 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.883504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98" event={"ID":"ab6700df-741e-4531-855d-d09717ed35d8","Type":"ContainerStarted","Data":"9792bec990a4dad19f49dd8bd65daa15a44c98edf6daa6840ba86fd7dff91e29"}
Jul 01 18:38:57.884782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.884757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"36c2538943ba6567edf3269d7690e13e796c1efba6f9bfe3bb79b88ee0600796"}
Jul 01 18:38:57.889955 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.889898    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:38:57.914240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.914202    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-zhm97" podStartSLOduration=1.662706416 podStartE2EDuration="2.914187917s" totalImagesPullingTime="1.251481501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:57.913227312 +0000 UTC m=+95.928806856" watchObservedRunningTime="2026-07-01 18:38:57.914187917 +0000 UTC m=+95.929767462"
Jul 01 18:38:57.942754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:57.942711    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-8cb98" podStartSLOduration=1.596008813 podStartE2EDuration="2.942697919s" totalImagesPullingTime="1.346689106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:57.941076565 +0000 UTC m=+95.956656139" watchObservedRunningTime="2026-07-01 18:38:57.942697919 +0000 UTC m=+95.958277505"
Jul 01 18:38:58.890394 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:58.890321    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a7be955-7677-4339-8d73-7ced4799a754" containerID="d2b8ce136291b6b4e8ca2d0400cbfa75d2bd494ea76cd13964af4c4f5a5fb8e3" exitCode=0
Jul 01 18:38:58.890742 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:58.890417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpcvc" event={"ID":"2a7be955-7677-4339-8d73-7ced4799a754","Type":"ContainerDied","Data":"d2b8ce136291b6b4e8ca2d0400cbfa75d2bd494ea76cd13964af4c4f5a5fb8e3"}
Jul 01 18:38:58.892013 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:58.891987    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b" exitCode=0
Jul 01 18:38:58.892105 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:58.892022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"}
Jul 01 18:38:59.898630 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:59.898586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpcvc" event={"ID":"2a7be955-7677-4339-8d73-7ced4799a754","Type":"ContainerStarted","Data":"22dc3df7346bd235cba92c4d0611decc4816dbd817b8407cea2e42968242b6f7"}
Jul 01 18:38:59.899047 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:59.898675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpcvc" event={"ID":"2a7be955-7677-4339-8d73-7ced4799a754","Type":"ContainerStarted","Data":"b69f426c0c6cf3948818190a7203e35a22d86eb20d4c7147abecca88c8f59a02"}
Jul 01 18:38:59.948417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:38:59.948367    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bpcvc" podStartSLOduration=3.39079273 podStartE2EDuration="4.948353063s" totalImagesPullingTime="1.486006228s" totalInitContainerRuntime="71.554105ms" isStatefulPod=true podCreationTimestamp="2026-07-01 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:59.947113514 +0000 UTC m=+97.962693062" watchObservedRunningTime="2026-07-01 18:38:59.948353063 +0000 UTC m=+97.963932621"
Jul 01 18:39:00.273026 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.273005    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:39:00.374175 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.374147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:39:00.374268 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.374214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:00.376385 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.376358    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fae58e6-d31b-4218-a13e-826368c57ed1-metrics-tls\") pod \"dns-default-s4lpv\" (UID: \"7fae58e6-d31b-4218-a13e-826368c57ed1\") " pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:39:00.376497 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.376480    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"image-registry-58b69d6978-m4vw7\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") " pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:00.635606 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.635582    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qpglp\""
Jul 01 18:39:00.643099 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.643067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:00.654775 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.654137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-555k5\""
Jul 01 18:39:00.658244 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.658169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:39:00.816837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.816811    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:39:00.820508 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:39:00.820477    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea0e6b4_3c99_4716_b31b_746a9eb671c4.slice/crio-ea2d88f7f5fb2f4e07f96ece9957e72e21e43abaa401ba97f8dd9b9f78ab4ade WatchSource:0}: Error finding container ea2d88f7f5fb2f4e07f96ece9957e72e21e43abaa401ba97f8dd9b9f78ab4ade: Status 404 returned error can't find the container with id ea2d88f7f5fb2f4e07f96ece9957e72e21e43abaa401ba97f8dd9b9f78ab4ade
Jul 01 18:39:00.860670 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.860604    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s4lpv"]
Jul 01 18:39:00.862505 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:39:00.862477    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae58e6_d31b_4218_a13e_826368c57ed1.slice/crio-e08cd15b9683466a5d5dedeebbdd99493c216ba06292d7a1cd333bfa2d0a2874 WatchSource:0}: Error finding container e08cd15b9683466a5d5dedeebbdd99493c216ba06292d7a1cd333bfa2d0a2874: Status 404 returned error can't find the container with id e08cd15b9683466a5d5dedeebbdd99493c216ba06292d7a1cd333bfa2d0a2874
Jul 01 18:39:00.905855 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.905789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s4lpv" event={"ID":"7fae58e6-d31b-4218-a13e-826368c57ed1","Type":"ContainerStarted","Data":"e08cd15b9683466a5d5dedeebbdd99493c216ba06292d7a1cd333bfa2d0a2874"}
Jul 01 18:39:00.907577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.907554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" event={"ID":"2ea0e6b4-3c99-4716-b31b-746a9eb671c4","Type":"ContainerStarted","Data":"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"}
Jul 01 18:39:00.907674 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.907599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" event={"ID":"2ea0e6b4-3c99-4716-b31b-746a9eb671c4","Type":"ContainerStarted","Data":"ea2d88f7f5fb2f4e07f96ece9957e72e21e43abaa401ba97f8dd9b9f78ab4ade"}
Jul 01 18:39:00.907717 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.907674    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:00.911349 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.911324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"}
Jul 01 18:39:00.911446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.911397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"}
Jul 01 18:39:00.911446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.911440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"}
Jul 01 18:39:00.911561 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.911479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"}
Jul 01 18:39:00.911561 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.911514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"}
Jul 01 18:39:00.939771 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:00.939727    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" podStartSLOduration=99.939714103 podStartE2EDuration="1m39.939714103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:00.938754774 +0000 UTC m=+98.954334319" watchObservedRunningTime="2026-07-01 18:39:00.939714103 +0000 UTC m=+98.955293646"
Jul 01 18:39:01.925744 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:01.925703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerStarted","Data":"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"}
Jul 01 18:39:01.982846 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:01.982786    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.112492928 podStartE2EDuration="5.982768382s" totalImagesPullingTime="2.855539542s" totalInitContainerRuntime="14.735912ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:01.980096533 +0000 UTC m=+99.995676101" watchObservedRunningTime="2026-07-01 18:39:01.982768382 +0000 UTC m=+99.998347926"
Jul 01 18:39:02.201431 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.201345    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:02.205517 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.205493    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.208546 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.208515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:39:02.208660 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.208518    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:39:02.208660 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.208525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:39:02.208910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.208892    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 18:39:02.209666 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.209411    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:39:02.209666 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.209487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sxn6t\""
Jul 01 18:39:02.209803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.209710    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:39:02.209803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.209749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:39:02.210393 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.210094    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-81h42c6odlob7\""
Jul 01 18:39:02.210393 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.210127    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:39:02.210393 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.210173    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:39:02.210393 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.210245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:39:02.210883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.210861    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:39:02.211348 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.211096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:39:02.211446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.211421    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:39:02.213913 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.213889    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:39:02.216864 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.216449    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:39:02.227510 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.227477    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:02.392181 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392384 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392576 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392576 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392576 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392576 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt2v\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.392763 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.393006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.392982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt2v\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.494667 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.494992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495226 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.495649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.495321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.497340 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.497311    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.498777 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.498464    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.498989 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.498965    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.499578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.499848    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.499862    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.499985    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.500046    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.500227    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500364 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.500328    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500751 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.500517    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.500956 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.500936    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.501618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.501594    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.501846 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.501825    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.502299 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.502261    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.502906 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.502859    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.503400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.503361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.505560 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.503595    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.505560 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.504749    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.512811 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.512769    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt2v\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v\") pod \"prometheus-k8s-0\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.521629 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.521606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:02.986206 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:02.986174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:03.118460 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:39:03.118427    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ff1bc_289a_4b98_8360_0f354c5af7f9.slice/crio-affdb8538e58109563e0dfd069cb54d4d44872b3c474af7612565f2759beefbf WatchSource:0}: Error finding container affdb8538e58109563e0dfd069cb54d4d44872b3c474af7612565f2759beefbf: Status 404 returned error can't find the container with id affdb8538e58109563e0dfd069cb54d4d44872b3c474af7612565f2759beefbf
Jul 01 18:39:03.759526 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.759461    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2kb8s"
Jul 01 18:39:03.937843 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.937817    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d" exitCode=0
Jul 01 18:39:03.938019 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.937903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"}
Jul 01 18:39:03.938019 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.937975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"affdb8538e58109563e0dfd069cb54d4d44872b3c474af7612565f2759beefbf"}
Jul 01 18:39:03.939897 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.939872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s4lpv" event={"ID":"7fae58e6-d31b-4218-a13e-826368c57ed1","Type":"ContainerStarted","Data":"61db831556684d789c8d22280668f30574740ef926dd0a99275eecd56b83b525"}
Jul 01 18:39:03.940043 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.939918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s4lpv" event={"ID":"7fae58e6-d31b-4218-a13e-826368c57ed1","Type":"ContainerStarted","Data":"e7eca3949f0bb11d9817f243d968277b5be70636821230a5216051c27aa5604b"}
Jul 01 18:39:03.940043 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:03.940002    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:39:04.004521 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:04.004486    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s4lpv" podStartSLOduration=65.717081176 podStartE2EDuration="1m8.00447304s" totalImagesPullingTime="2.287391864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:04.003774839 +0000 UTC m=+102.019354384" watchObservedRunningTime="2026-07-01 18:39:04.00447304 +0000 UTC m=+102.020052583"
Jul 01 18:39:05.528183 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:05.528149    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:39:05.977249 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:05.977218    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:39:05.999711 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:05.999681    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:39:05.999858 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:05.999770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.014582 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.014554    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 18:39:06.022479 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022479 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpc5s\" (UniqueName: \"kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022636 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022636 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022718 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022718 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.022795 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.022719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123298 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123431 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123431 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123560 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123560 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc5s\" (UniqueName: \"kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123659 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.123659 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.123593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.124155 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.124109    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.124278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.124242    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.124391 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.124300    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.124391 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.124363    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.126153 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.126130    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.126362 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.126346    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.137493 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.137471    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc5s\" (UniqueName: \"kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s\") pod \"console-64f4b456f5-jndhn\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") " pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:06.317398 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:06.317326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:07.732675 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:07.732648    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:39:07.847321 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:39:07.847261    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88384853_f81c_4b04_864f_ceebc608477d.slice/crio-9e90342104763870b259d1b0e022bf49bec13152dbcf4c0e4846a244fbae2e5f WatchSource:0}: Error finding container 9e90342104763870b259d1b0e022bf49bec13152dbcf4c0e4846a244fbae2e5f: Status 404 returned error can't find the container with id 9e90342104763870b259d1b0e022bf49bec13152dbcf4c0e4846a244fbae2e5f
Jul 01 18:39:07.961508 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:07.961477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"}
Jul 01 18:39:07.963317 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:07.963244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f4b456f5-jndhn" event={"ID":"88384853-f81c-4b04-864f-ceebc608477d","Type":"ContainerStarted","Data":"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"}
Jul 01 18:39:07.963317 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:07.963304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f4b456f5-jndhn" event={"ID":"88384853-f81c-4b04-864f-ceebc608477d","Type":"ContainerStarted","Data":"9e90342104763870b259d1b0e022bf49bec13152dbcf4c0e4846a244fbae2e5f"}
Jul 01 18:39:07.996179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:07.996127    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64f4b456f5-jndhn" podStartSLOduration=2.996107778 podStartE2EDuration="2.996107778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:07.994561307 +0000 UTC m=+106.010140877" watchObservedRunningTime="2026-07-01 18:39:07.996107778 +0000 UTC m=+106.011687323"
Jul 01 18:39:08.971868 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:08.971833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"}
Jul 01 18:39:10.985057 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:10.985018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"}
Jul 01 18:39:10.985547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:10.985095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"}
Jul 01 18:39:10.985547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:10.985135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"}
Jul 01 18:39:10.985547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:10.985187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerStarted","Data":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"}
Jul 01 18:39:11.038578 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:11.038534    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.999285138 podStartE2EDuration="9.03851929s" totalImagesPullingTime="6.01474048s" totalInitContainerRuntime="24.493672ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:11.038145489 +0000 UTC m=+109.053725096" watchObservedRunningTime="2026-07-01 18:39:11.03851929 +0000 UTC m=+109.054098879"
Jul 01 18:39:12.522146 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:12.522117    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:13.948591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:13.948562    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s4lpv"
Jul 01 18:39:15.535520 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:15.535494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:16.318271 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:16.318235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:16.318271 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:16.318311    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:16.322953 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:16.322930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:17.016140 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:17.016112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:39:25.931242 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:25.931171    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d95b95bb-ghv6d" podUID="6ccf9274-6105-4faf-8e15-258cec365b67" containerName="console" containerID="cri-o://12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730" gracePeriod=15
Jul 01 18:39:26.705363 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.705341    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:39:26.793758 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.793758 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.793953 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.793953 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.793953 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn595\" (UniqueName: \"kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.793953 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.793913    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config\") pod \"6ccf9274-6105-4faf-8e15-258cec365b67\" (UID: \"6ccf9274-6105-4faf-8e15-258cec365b67\") "
Jul 01 18:39:26.794362 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.794232    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca" (OuterVolumeSpecName: "service-ca") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:26.794492 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.794357    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:26.794492 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.794396    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config" (OuterVolumeSpecName: "console-config") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:26.795975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.795952    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:26.796070 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.796030    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:26.796328 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.796268    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595" (OuterVolumeSpecName: "kube-api-access-vn595") pod "6ccf9274-6105-4faf-8e15-258cec365b67" (UID: "6ccf9274-6105-4faf-8e15-258cec365b67"). InnerVolumeSpecName "kube-api-access-vn595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:26.894905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894885    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-oauth-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:26.894985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894907    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ccf9274-6105-4faf-8e15-258cec365b67-console-serving-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:26.894985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894917    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-oauth-serving-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:26.894985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894927    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-service-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:26.894985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894936    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn595\" (UniqueName: \"kubernetes.io/projected/6ccf9274-6105-4faf-8e15-258cec365b67-kube-api-access-vn595\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:26.894985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:26.894946    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ccf9274-6105-4faf-8e15-258cec365b67-console-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:27.053386 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.053357    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ccf9274-6105-4faf-8e15-258cec365b67" containerID="12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730" exitCode=0
Jul 01 18:39:27.053669 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.053401    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d95b95bb-ghv6d"
Jul 01 18:39:27.053669 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.053447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d95b95bb-ghv6d" event={"ID":"6ccf9274-6105-4faf-8e15-258cec365b67","Type":"ContainerDied","Data":"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"}
Jul 01 18:39:27.053669 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.053489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d95b95bb-ghv6d" event={"ID":"6ccf9274-6105-4faf-8e15-258cec365b67","Type":"ContainerDied","Data":"d9eae5053e0183f439914666fcda45950a604df7074932ce43de093aed486d14"}
Jul 01 18:39:27.054352 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.054339    2583 scope.go:117] "RemoveContainer" containerID="12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"
Jul 01 18:39:27.063063 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.063035    2583 scope.go:117] "RemoveContainer" containerID="12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"
Jul 01 18:39:27.063355 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:39:27.063334    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730\": container with ID starting with 12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730 not found: ID does not exist" containerID="12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"
Jul 01 18:39:27.063414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.063363    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730"} err="failed to get container status \"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730\": rpc error: code = NotFound desc = could not find container \"12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730\": container with ID starting with 12462ee6e8f358ee64da816675eb46200fbd15eef9e08970941aefd1c306b730 not found: ID does not exist"
Jul 01 18:39:27.085866 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.085840    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:39:27.101260 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:27.101240    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d95b95bb-ghv6d"]
Jul 01 18:39:28.366339 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:28.366309    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ccf9274-6105-4faf-8e15-258cec365b67" path="/var/lib/kubelet/pods/6ccf9274-6105-4faf-8e15-258cec365b67/volumes"
Jul 01 18:39:30.548247 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.548206    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" podUID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" containerName="registry" containerID="cri-o://d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7" gracePeriod=30
Jul 01 18:39:30.789776 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.789754    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:30.927617 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvsjg\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.927754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.927754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.927754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.927754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927714    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.927754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.928004 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927765    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.928004 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.927797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration\") pod \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\" (UID: \"2ea0e6b4-3c99-4716-b31b-746a9eb671c4\") "
Jul 01 18:39:30.928211 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.928183    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:30.928278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.928199    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:30.930133 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.930096    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg" (OuterVolumeSpecName: "kube-api-access-bvsjg") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "kube-api-access-bvsjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:30.930240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.930148    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:30.930341 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.930316    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:30.930402 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.930338    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:30.930451 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.930431    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:30.936041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:30.936018    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2ea0e6b4-3c99-4716-b31b-746a9eb671c4" (UID: "2ea0e6b4-3c99-4716-b31b-746a9eb671c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:39:31.028809 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028779    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-installation-pull-secrets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028809 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028804    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028821    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-image-registry-private-configuration\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028834    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvsjg\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-kube-api-access-bvsjg\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028847    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-registry-certificates\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028859    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-ca-trust-extracted\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028871    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-bound-sa-token\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.028945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.028883    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ea0e6b4-3c99-4716-b31b-746a9eb671c4-trusted-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:39:31.071443 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.071412    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" containerID="d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7" exitCode=0
Jul 01 18:39:31.071533 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.071462    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7"
Jul 01 18:39:31.071533 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.071498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" event={"ID":"2ea0e6b4-3c99-4716-b31b-746a9eb671c4","Type":"ContainerDied","Data":"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"}
Jul 01 18:39:31.071601 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.071552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b69d6978-m4vw7" event={"ID":"2ea0e6b4-3c99-4716-b31b-746a9eb671c4","Type":"ContainerDied","Data":"ea2d88f7f5fb2f4e07f96ece9957e72e21e43abaa401ba97f8dd9b9f78ab4ade"}
Jul 01 18:39:31.071601 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.071573    2583 scope.go:117] "RemoveContainer" containerID="d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"
Jul 01 18:39:31.080231 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.080215    2583 scope.go:117] "RemoveContainer" containerID="d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"
Jul 01 18:39:31.080495 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:39:31.080477    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7\": container with ID starting with d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7 not found: ID does not exist" containerID="d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"
Jul 01 18:39:31.080558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.080502    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7"} err="failed to get container status \"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7\": rpc error: code = NotFound desc = could not find container \"d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7\": container with ID starting with d314292356632eb1626ac377a728a756f0b95f4a866a6895e2b57ae0e9e75da7 not found: ID does not exist"
Jul 01 18:39:31.096539 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.096518    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:39:31.105633 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.105614    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58b69d6978-m4vw7"]
Jul 01 18:39:31.230635 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.230575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:39:31.232655 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.232640    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bffaf8-f1aa-48f2-a03f-e2161a1e7e86-metrics-certs\") pod \"network-metrics-daemon-4pb8p\" (UID: \"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86\") " pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:39:31.387114 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.387090    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t4jbq\""
Jul 01 18:39:31.394524 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.394505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4pb8p"
Jul 01 18:39:31.522519 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:31.522496    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4pb8p"]
Jul 01 18:39:31.524644 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:39:31.524617    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bffaf8_f1aa_48f2_a03f_e2161a1e7e86.slice/crio-b72e26853160da8293b6e2f32d18e5b8925beacaae56b69237085a7f7639a0cd WatchSource:0}: Error finding container b72e26853160da8293b6e2f32d18e5b8925beacaae56b69237085a7f7639a0cd: Status 404 returned error can't find the container with id b72e26853160da8293b6e2f32d18e5b8925beacaae56b69237085a7f7639a0cd
Jul 01 18:39:32.077514 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:32.077481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4pb8p" event={"ID":"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86","Type":"ContainerStarted","Data":"b72e26853160da8293b6e2f32d18e5b8925beacaae56b69237085a7f7639a0cd"}
Jul 01 18:39:32.367797 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:32.367771    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" path="/var/lib/kubelet/pods/2ea0e6b4-3c99-4716-b31b-746a9eb671c4/volumes"
Jul 01 18:39:33.084634 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:33.084597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4pb8p" event={"ID":"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86","Type":"ContainerStarted","Data":"d404ef4038c761a7b8d1043c55875c692871dfdbf4712e326364ab14a3936ce4"}
Jul 01 18:39:33.084965 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:33.084695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4pb8p" event={"ID":"03bffaf8-f1aa-48f2-a03f-e2161a1e7e86","Type":"ContainerStarted","Data":"56114f565a053f41a77777fea2646a72d291ea34e2384d90737c71b2dee891b7"}
Jul 01 18:39:33.108704 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:39:33.108658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4pb8p" podStartSLOduration=130.010519607 podStartE2EDuration="2m11.108643107s" totalImagesPullingTime="1.0981235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:33.107278527 +0000 UTC m=+131.122858073" watchObservedRunningTime="2026-07-01 18:39:33.108643107 +0000 UTC m=+131.124222704"
Jul 01 18:40:02.522601 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:02.522573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:02.542010 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:02.541975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:03.218905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:03.218879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.274965    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.275514    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="alertmanager" containerID="cri-o://cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d" gracePeriod=120
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.275797    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy" containerID="cri-o://c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853" gracePeriod=120
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.275897    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-web" containerID="cri-o://d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46" gracePeriod=120
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.275959    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="prom-label-proxy" containerID="cri-o://ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae" gracePeriod=120
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.276017    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="config-reloader" containerID="cri-o://a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe" gracePeriod=120
Jul 01 18:40:16.276900 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:16.276070    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-metric" containerID="cri-o://8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4" gracePeriod=120
Jul 01 18:40:17.261484 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261454    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae" exitCode=0
Jul 01 18:40:17.261484 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261479    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853" exitCode=0
Jul 01 18:40:17.261484 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261485    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe" exitCode=0
Jul 01 18:40:17.261484 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261492    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d" exitCode=0
Jul 01 18:40:17.261729 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"}
Jul 01 18:40:17.261729 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"}
Jul 01 18:40:17.261729 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"}
Jul 01 18:40:17.261729 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.261641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"}
Jul 01 18:40:17.528944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.528918    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592640    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592667    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97wtb\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592813    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592859    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592890    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.593461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.592982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle\") pod \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\" (UID: \"e10a91c8-fadb-4427-a8a2-67bc1d83da04\") "
Jul 01 18:40:17.594686 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.594389    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:17.598796 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.598507    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:17.598796 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.598686    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.600254 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.600195    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:17.608746 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.607804    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:17.608746 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.607934    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.609157 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.609099    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.609624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.609586    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.611070 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.611029    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out" (OuterVolumeSpecName: "config-out") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:17.611457 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.611264    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume" (OuterVolumeSpecName: "config-volume") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.618032 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.617986    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb" (OuterVolumeSpecName: "kube-api-access-97wtb") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "kube-api-access-97wtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:17.619242 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.619214    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.626392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.626370    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config" (OuterVolumeSpecName: "web-config") pod "e10a91c8-fadb-4427-a8a2-67bc1d83da04" (UID: "e10a91c8-fadb-4427-a8a2-67bc1d83da04"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:17.693885 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693864    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-cluster-tls-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.693885 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693885    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693894    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-main-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693904    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wtb\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-kube-api-access-97wtb\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693913    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-main-db\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693922    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693931    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-out\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693939    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-metrics-client-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693947    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-config-volume\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693955    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-web-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693962    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10a91c8-fadb-4427-a8a2-67bc1d83da04-tls-assets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693970    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e10a91c8-fadb-4427-a8a2-67bc1d83da04-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:17.694006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:17.693978    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10a91c8-fadb-4427-a8a2-67bc1d83da04-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:18.268279 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268249    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4" exitCode=0
Jul 01 18:40:18.268279 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268272    2583 generic.go:356] "Generic (PLEG): container finished" podID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerID="d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46" exitCode=0
Jul 01 18:40:18.268452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"}
Jul 01 18:40:18.268452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"}
Jul 01 18:40:18.268452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268338    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.268452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e10a91c8-fadb-4427-a8a2-67bc1d83da04","Type":"ContainerDied","Data":"36c2538943ba6567edf3269d7690e13e796c1efba6f9bfe3bb79b88ee0600796"}
Jul 01 18:40:18.268586 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.268467    2583 scope.go:117] "RemoveContainer" containerID="ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"
Jul 01 18:40:18.275798 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.275781    2583 scope.go:117] "RemoveContainer" containerID="8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"
Jul 01 18:40:18.282629 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.282612    2583 scope.go:117] "RemoveContainer" containerID="c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"
Jul 01 18:40:18.289266 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.289250    2583 scope.go:117] "RemoveContainer" containerID="d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"
Jul 01 18:40:18.295265 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.295248    2583 scope.go:117] "RemoveContainer" containerID="a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"
Jul 01 18:40:18.300567 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.300546    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:18.303003 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.302988    2583 scope.go:117] "RemoveContainer" containerID="cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"
Jul 01 18:40:18.309643 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.309623    2583 scope.go:117] "RemoveContainer" containerID="15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"
Jul 01 18:40:18.313335 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.313315    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:18.316584 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.316569    2583 scope.go:117] "RemoveContainer" containerID="ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"
Jul 01 18:40:18.316813 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.316795    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae\": container with ID starting with ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae not found: ID does not exist" containerID="ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"
Jul 01 18:40:18.316865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.316820    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"} err="failed to get container status \"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae\": rpc error: code = NotFound desc = could not find container \"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae\": container with ID starting with ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae not found: ID does not exist"
Jul 01 18:40:18.316865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.316835    2583 scope.go:117] "RemoveContainer" containerID="8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"
Jul 01 18:40:18.317061 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.317045    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4\": container with ID starting with 8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4 not found: ID does not exist" containerID="8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"
Jul 01 18:40:18.317108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317065    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"} err="failed to get container status \"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4\": rpc error: code = NotFound desc = could not find container \"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4\": container with ID starting with 8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4 not found: ID does not exist"
Jul 01 18:40:18.317108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317081    2583 scope.go:117] "RemoveContainer" containerID="c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"
Jul 01 18:40:18.317248 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.317234    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853\": container with ID starting with c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853 not found: ID does not exist" containerID="c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"
Jul 01 18:40:18.317301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317252    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"} err="failed to get container status \"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853\": rpc error: code = NotFound desc = could not find container \"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853\": container with ID starting with c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853 not found: ID does not exist"
Jul 01 18:40:18.317301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317264    2583 scope.go:117] "RemoveContainer" containerID="d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"
Jul 01 18:40:18.317474 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.317460    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46\": container with ID starting with d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46 not found: ID does not exist" containerID="d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"
Jul 01 18:40:18.317509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317476    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"} err="failed to get container status \"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46\": rpc error: code = NotFound desc = could not find container \"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46\": container with ID starting with d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46 not found: ID does not exist"
Jul 01 18:40:18.317509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317488    2583 scope.go:117] "RemoveContainer" containerID="a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"
Jul 01 18:40:18.317677 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.317663    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe\": container with ID starting with a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe not found: ID does not exist" containerID="a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"
Jul 01 18:40:18.317724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317677    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"} err="failed to get container status \"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe\": rpc error: code = NotFound desc = could not find container \"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe\": container with ID starting with a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe not found: ID does not exist"
Jul 01 18:40:18.317724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317688    2583 scope.go:117] "RemoveContainer" containerID="cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"
Jul 01 18:40:18.317850 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.317837    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d\": container with ID starting with cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d not found: ID does not exist" containerID="cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"
Jul 01 18:40:18.317890 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317851    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"} err="failed to get container status \"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d\": rpc error: code = NotFound desc = could not find container \"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d\": container with ID starting with cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d not found: ID does not exist"
Jul 01 18:40:18.317890 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.317861    2583 scope.go:117] "RemoveContainer" containerID="15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"
Jul 01 18:40:18.318079 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:18.318066    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b\": container with ID starting with 15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b not found: ID does not exist" containerID="15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"
Jul 01 18:40:18.318122 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318080    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"} err="failed to get container status \"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b\": rpc error: code = NotFound desc = could not find container \"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b\": container with ID starting with 15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b not found: ID does not exist"
Jul 01 18:40:18.318122 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318091    2583 scope.go:117] "RemoveContainer" containerID="ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"
Jul 01 18:40:18.318279 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318264    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae"} err="failed to get container status \"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae\": rpc error: code = NotFound desc = could not find container \"ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae\": container with ID starting with ac1565c3a47fb2b2167f09fc24f45c9eceeb5ecec67e62b0b4db7d57b2f4a3ae not found: ID does not exist"
Jul 01 18:40:18.318279 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318278    2583 scope.go:117] "RemoveContainer" containerID="8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"
Jul 01 18:40:18.318477 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318462    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4"} err="failed to get container status \"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4\": rpc error: code = NotFound desc = could not find container \"8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4\": container with ID starting with 8d8daf6453302892b3cbccd953f0ea4c84998bf62a815a647c2c6243d367a6b4 not found: ID does not exist"
Jul 01 18:40:18.318519 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318476    2583 scope.go:117] "RemoveContainer" containerID="c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"
Jul 01 18:40:18.318668 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318654    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853"} err="failed to get container status \"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853\": rpc error: code = NotFound desc = could not find container \"c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853\": container with ID starting with c5a9a2d152721182ebcefeaf775c20fa10859b5fc414c918ae9992947b749853 not found: ID does not exist"
Jul 01 18:40:18.318668 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318667    2583 scope.go:117] "RemoveContainer" containerID="d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"
Jul 01 18:40:18.318836 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318822    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46"} err="failed to get container status \"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46\": rpc error: code = NotFound desc = could not find container \"d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46\": container with ID starting with d5d5db42c03eed805ed297d46f5d8f5ab843c58438bc067196f4fb2b8fcace46 not found: ID does not exist"
Jul 01 18:40:18.318876 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318836    2583 scope.go:117] "RemoveContainer" containerID="a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"
Jul 01 18:40:18.319000 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.318981    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe"} err="failed to get container status \"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe\": rpc error: code = NotFound desc = could not find container \"a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe\": container with ID starting with a67b1997d9b055777b90908330300a297d0cd54d8ab24352dbca89172abf37fe not found: ID does not exist"
Jul 01 18:40:18.319041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.319001    2583 scope.go:117] "RemoveContainer" containerID="cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"
Jul 01 18:40:18.319193 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.319175    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d"} err="failed to get container status \"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d\": rpc error: code = NotFound desc = could not find container \"cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d\": container with ID starting with cb85f9a3a9fca4900bd02424fd1721d6295dc06b47dcf3d23f00182a4e39ca5d not found: ID does not exist"
Jul 01 18:40:18.319252 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.319195    2583 scope.go:117] "RemoveContainer" containerID="15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"
Jul 01 18:40:18.319453 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.319438    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b"} err="failed to get container status \"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b\": rpc error: code = NotFound desc = could not find container \"15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b\": container with ID starting with 15c4cb0d0c0e56e59e976f24446ec3cbe68702326f5e11c98f44033c1e52521b not found: ID does not exist"
Jul 01 18:40:18.366703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.366683    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" path="/var/lib/kubelet/pods/e10a91c8-fadb-4427-a8a2-67bc1d83da04/volumes"
Jul 01 18:40:18.382880 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.382857    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:18.383188 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383172    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="init-config-reloader"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383191    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="init-config-reloader"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383211    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" containerName="registry"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383220    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" containerName="registry"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383231    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-metric"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383240    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-metric"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383255    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccf9274-6105-4faf-8e15-258cec365b67" containerName="console"
Jul 01 18:40:18.383263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383264    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccf9274-6105-4faf-8e15-258cec365b67" containerName="console"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383279    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-web"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383311    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-web"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383323    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383333    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383344    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="config-reloader"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383351    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="config-reloader"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383388    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="alertmanager"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383397    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="alertmanager"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383416    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="prom-label-proxy"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="prom-label-proxy"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383494    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea0e6b4-3c99-4716-b31b-746a9eb671c4" containerName="registry"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383508    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="config-reloader"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383518    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383526    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccf9274-6105-4faf-8e15-258cec365b67" containerName="console"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383534    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-metric"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383543    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="alertmanager"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383551    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="kube-rbac-proxy-web"
Jul 01 18:40:18.383614 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.383558    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10a91c8-fadb-4427-a8a2-67bc1d83da04" containerName="prom-label-proxy"
Jul 01 18:40:18.388663 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.388647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.391765 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.391726    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 18:40:18.391869 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.391849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 18:40:18.392717 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.392700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 18:40:18.392796 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.392742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 18:40:18.392865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.392819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lvkv5\""
Jul 01 18:40:18.392865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.392859    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 18:40:18.392996 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.392981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 18:40:18.393074 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.393056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 18:40:18.393132 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.393063    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 18:40:18.400346 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.400330    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 18:40:18.423657 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.423634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:18.498862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.498842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.498958 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.498872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.498958 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.498892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.498958 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.498910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.498958 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.498949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499084 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqmx\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-kube-api-access-9mqmx\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499084 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-out\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499084 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-web-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499084 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499209 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499209 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499209 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.499321 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.499218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600316 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600316 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqmx\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-kube-api-access-9mqmx\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-out\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-web-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.600654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.601173 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.600665    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.601173 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.601083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.601861 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.601833    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3af68a2-cc04-4c8d-a79d-386e00e2e130-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.603446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.603411    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.603580 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.603555    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.603698 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.603645    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.603698 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.603645    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.603698 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.603683    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.604047 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.604021    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3af68a2-cc04-4c8d-a79d-386e00e2e130-config-out\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.604137 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.604043    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.604329 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.604313    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-web-config\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.605501 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.605477    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3af68a2-cc04-4c8d-a79d-386e00e2e130-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.611567 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.611544    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqmx\" (UniqueName: \"kubernetes.io/projected/a3af68a2-cc04-4c8d-a79d-386e00e2e130-kube-api-access-9mqmx\") pod \"alertmanager-main-0\" (UID: \"a3af68a2-cc04-4c8d-a79d-386e00e2e130\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.698197 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.698167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:40:18.841489 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:18.841388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:40:18.843560 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:40:18.843536    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3af68a2_cc04_4c8d_a79d_386e00e2e130.slice/crio-45dbe8aa61d6a7666d6e663918d64788c48bd16e9e678ff6607df1065f815964 WatchSource:0}: Error finding container 45dbe8aa61d6a7666d6e663918d64788c48bd16e9e678ff6607df1065f815964: Status 404 returned error can't find the container with id 45dbe8aa61d6a7666d6e663918d64788c48bd16e9e678ff6607df1065f815964
Jul 01 18:40:19.274943 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:19.274913    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3af68a2-cc04-4c8d-a79d-386e00e2e130" containerID="dfaad34826ec2fa112f6a6f6bfe73832cb42b7c27258e58a57dbd4ac217eede6" exitCode=0
Jul 01 18:40:19.275128 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:19.274992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerDied","Data":"dfaad34826ec2fa112f6a6f6bfe73832cb42b7c27258e58a57dbd4ac217eede6"}
Jul 01 18:40:19.275128 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:19.275076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"45dbe8aa61d6a7666d6e663918d64788c48bd16e9e678ff6607df1065f815964"}
Jul 01 18:40:20.289453 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"d141a3678b013711c9bd27f6e5b467ae8157244ebfca4a979227affc2321c089"}
Jul 01 18:40:20.289889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"115ab8ffd7cb84926a07f090d5493f53e5e2055f42110ef0675dd57ad269b972"}
Jul 01 18:40:20.289889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"3917fcce8100f1e407f0c29e826142fdda9dcac39c97a5151ec12681bed1ee08"}
Jul 01 18:40:20.289889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"cbcfdda669da01d5d5e38329480ba4909d2d3bbde568613ad3b5f15ee7f4335a"}
Jul 01 18:40:20.289889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"0a06f0f02075fbf9a2eb3b8947239d03fad6c25ecd9f0d8af79f0f03eb82c736"}
Jul 01 18:40:20.289889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.289640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3af68a2-cc04-4c8d-a79d-386e00e2e130","Type":"ContainerStarted","Data":"87da9e4e12bb74dbb6f630353fda1ec3ec189b6278afd6a8b4b3a214d2d740c8"}
Jul 01 18:40:20.329890 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.329826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.315003512 podStartE2EDuration="2.329801994s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.798482ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:40:20.326965749 +0000 UTC m=+178.342545295" watchObservedRunningTime="2026-07-01 18:40:20.329801994 +0000 UTC m=+178.345381540"
Jul 01 18:40:20.672196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672169    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:20.672518 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672457    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="prometheus" containerID="cri-o://085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d" gracePeriod=600
Jul 01 18:40:20.672518 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672484    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-web" containerID="cri-o://7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164" gracePeriod=600
Jul 01 18:40:20.672518 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672506    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy" containerID="cri-o://0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000" gracePeriod=600
Jul 01 18:40:20.672744 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672486    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="thanos-sidecar" containerID="cri-o://ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36" gracePeriod=600
Jul 01 18:40:20.672744 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672490    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449" gracePeriod=600
Jul 01 18:40:20.672744 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.672512    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="config-reloader" containerID="cri-o://b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37" gracePeriod=600
Jul 01 18:40:20.909111 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:20.909081    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.023839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.023839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.023839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.023839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.023986    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024045    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024125    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024204 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024227    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024270    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024280    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024409    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbt2v\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v\") pod \"067ff1bc-289a-4b98-8360-0f354c5af7f9\" (UID: \"067ff1bc-289a-4b98-8360-0f354c5af7f9\") "
Jul 01 18:40:21.024680 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024606    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.025026 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024751    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.025026 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.024773    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-metrics-client-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.025140 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.025054    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.026562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.026450    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config" (OuterVolumeSpecName: "config") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.027125 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.027087    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v" (OuterVolumeSpecName: "kube-api-access-kbt2v") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "kube-api-access-kbt2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:21.027217 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.027200    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.027304 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.027248    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.027717 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.027668    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.028174 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028142    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:21.028387 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028354    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.028485 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028460    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:21.028795 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028767    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:21.028888 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028791    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.028888 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.028825    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.029112 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.029075    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.029219 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.029156    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.029331 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.029308    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.029414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.029403    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.029859 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.029839    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out" (OuterVolumeSpecName: "config-out") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:21.038237 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.038214    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config" (OuterVolumeSpecName: "web-config") pod "067ff1bc-289a-4b98-8360-0f354c5af7f9" (UID: "067ff1bc-289a-4b98-8360-0f354c5af7f9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:21.125887 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125864    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-grpc-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.125887 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125888    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125899    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-config-out\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125908    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt2v\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-kube-api-access-kbt2v\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125917    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-kube-rbac-proxy\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125925    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067ff1bc-289a-4b98-8360-0f354c5af7f9-tls-assets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125933    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-metrics-client-certs\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125942    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125949    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-db\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125958    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125967    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-web-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125974    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125982    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.125998    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.126008    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.126016    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126433 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.126025    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/067ff1bc-289a-4b98-8360-0f354c5af7f9-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.126433 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.126033    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/067ff1bc-289a-4b98-8360-0f354c5af7f9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297127    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449" exitCode=0
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297148    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000" exitCode=0
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297156    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164" exitCode=0
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297162    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36" exitCode=0
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297168    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37" exitCode=0
Jul 01 18:40:21.297190 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297172    2583 generic.go:356] "Generic (PLEG): container finished" podID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d" exitCode=0
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297196    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297337    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"}
Jul 01 18:40:21.297721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.297445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"067ff1bc-289a-4b98-8360-0f354c5af7f9","Type":"ContainerDied","Data":"affdb8538e58109563e0dfd069cb54d4d44872b3c474af7612565f2759beefbf"}
Jul 01 18:40:21.306052 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.305946    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.312719 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.312704    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.319124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.319109    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.325212 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.325193    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.329591 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.329564    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:21.332306 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.332271    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.334814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.334795    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:21.339649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.339634    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.347181 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.347165    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.347442 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.347425    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.347507 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.347450    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.347507 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.347467    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.347734 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.347712    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.347782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.347742    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.347782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.347764    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.347994 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.347977    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.348036 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348008    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.348036 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348022    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.348253 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.348236    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.348314 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348268    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.348314 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348300    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.348538 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.348524    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.348593 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348541    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.348593 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348553    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.348805 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.348789    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.348843 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348809    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.348843 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.348822    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.349123 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:40:21.349058    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.349123 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349087    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.349123 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349108    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.349446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349373    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.349446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349389    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.349694 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349621    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.349694 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349637    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.349910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349843    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.349910 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.349862    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.350143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350069    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.350143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350082    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.350392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350272    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.350392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350311    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.350584 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350561    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.350627 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350588    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.350869 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.350964 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.350876    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.351165 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351135    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.351229 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351166    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.351478 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351456    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.351574 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351481    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.351750 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351733    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.351830 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.351753    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.352048 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352022    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.352108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352050    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.352338 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352313    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.352338 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352338    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.352609 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352585    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.352671 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352610    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.352884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.352971 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.352884    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.353136 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353115    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.353232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353137    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.353415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353389    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.353503 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353415    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.353724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353702    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.353796 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353724    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.354012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.353978    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.354012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354012    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.354334 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354273    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.354396 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354337    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.354593 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.354593 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354595    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.354834 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354817    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.354907 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.354837    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.355094 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355071    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.355180 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355095    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.355355 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355334    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.355427 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355356    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.355604 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355589    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.355670 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355605    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.355846 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355829    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.355907 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.355847    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.356136 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356112    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.356205 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356144    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.356424 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356404    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.356424 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356423    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.356684 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356662    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.356776 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356685    2583 scope.go:117] "RemoveContainer" containerID="780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"
Jul 01 18:40:21.356985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356956    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449"} err="failed to get container status \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": rpc error: code = NotFound desc = could not find container \"780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449\": container with ID starting with 780c6f2bb109c295aa471f566e4304dda4056dafa62406899129bdad514d6449 not found: ID does not exist"
Jul 01 18:40:21.356985 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.356978    2583 scope.go:117] "RemoveContainer" containerID="0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"
Jul 01 18:40:21.357234 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357214    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000"} err="failed to get container status \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": rpc error: code = NotFound desc = could not find container \"0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000\": container with ID starting with 0210a729cf9c0352fc56a64274b6ad16b9c3ca32f97750ce73b91921493e5000 not found: ID does not exist"
Jul 01 18:40:21.357324 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357237    2583 scope.go:117] "RemoveContainer" containerID="7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"
Jul 01 18:40:21.357549 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357520    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164"} err="failed to get container status \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": rpc error: code = NotFound desc = could not find container \"7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164\": container with ID starting with 7cb12e805982d1f534f9503f9d3985fd27842e5d55de20888455aafe53295164 not found: ID does not exist"
Jul 01 18:40:21.357549 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357539    2583 scope.go:117] "RemoveContainer" containerID="ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"
Jul 01 18:40:21.357814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357793    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36"} err="failed to get container status \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": rpc error: code = NotFound desc = could not find container \"ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36\": container with ID starting with ec55e96d1eef6910c5f31216e1beae67e98acec8e17b5d03db5ab2a77f1c2a36 not found: ID does not exist"
Jul 01 18:40:21.357814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.357812    2583 scope.go:117] "RemoveContainer" containerID="b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"
Jul 01 18:40:21.358071 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.358045    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37"} err="failed to get container status \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": rpc error: code = NotFound desc = could not find container \"b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37\": container with ID starting with b2521bafd2cfcf90f4b6c681236578e810e7a2cda84aaeef3caa23d707986b37 not found: ID does not exist"
Jul 01 18:40:21.358071 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.358071    2583 scope.go:117] "RemoveContainer" containerID="085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"
Jul 01 18:40:21.358344 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.358324    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d"} err="failed to get container status \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": rpc error: code = NotFound desc = could not find container \"085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d\": container with ID starting with 085f106c99fb66ccbea56ea793a9543b88ebd0aaf830ba38435b0824b079bd2d not found: ID does not exist"
Jul 01 18:40:21.358430 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.358346    2583 scope.go:117] "RemoveContainer" containerID="80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"
Jul 01 18:40:21.358596 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.358573    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d"} err="failed to get container status \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": rpc error: code = NotFound desc = could not find container \"80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d\": container with ID starting with 80ea05fb1b343a27c2a6394156b643eeb7ad1ad6db9ab82c724eb344ce75699d not found: ID does not exist"
Jul 01 18:40:21.372339 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372317    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:21.372580 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372570    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:21.372619 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372582    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:21.372619 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="prometheus"
Jul 01 18:40:21.372619 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372609    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="prometheus"
Jul 01 18:40:21.372619 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372615    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="config-reloader"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372620    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="config-reloader"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372629    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="init-config-reloader"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372634    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="init-config-reloader"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372641    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="thanos-sidecar"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372645    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="thanos-sidecar"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372650    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-web"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372655    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-web"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372661    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372666    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372712    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="config-reloader"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372719    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372726    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:21.372731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372732    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="prometheus"
Jul 01 18:40:21.373067 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372739    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="kube-rbac-proxy-web"
Jul 01 18:40:21.373067 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.372745    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" containerName="thanos-sidecar"
Jul 01 18:40:21.377971 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.377955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.382917 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.382886    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:40:21.382917 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.382911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:40:21.383102 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.382952    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:40:21.383102 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:40:21.383102 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.382888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:40:21.383272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383190    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:40:21.383272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383195    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:40:21.383272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383210    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:40:21.383272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383252    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-81h42c6odlob7\""
Jul 01 18:40:21.383488 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383202    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:40:21.383488 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383440    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 18:40:21.383585 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.383494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sxn6t\""
Jul 01 18:40:21.384892 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.384875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:40:21.385052 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.384991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:40:21.385378 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.385363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:40:21.387102 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.387081    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:40:21.388311 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.388277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:40:21.397599 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.397578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:21.428443 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-config-out\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428711 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428711 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428711 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str5b\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-kube-api-access-str5b\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428711 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428937 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.428937 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-web-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429003 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429003 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.428968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429074 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.429046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429110 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.429075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429151 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.429115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.429145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.429191 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.429175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529595 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-web-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529673 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529808 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529808 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529808 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.529944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.529984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-config-out\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str5b\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-kube-api-access-str5b\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530674 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530653    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.530819 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.530699    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.531279    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.531815    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.532267    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.532491    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.532753    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.532900    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.533318 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.533266    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.534236 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.534190    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/602fb654-5a74-4edc-8044-e1077c3c558c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.534340 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.534326    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.534947 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.534905    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/602fb654-5a74-4edc-8044-e1077c3c558c-config-out\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.534947 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.534922    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.535133 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.535104    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.535212 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.535107    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.535427 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.535408    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.535556 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.535536    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-web-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.535829 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.535812    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-config\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.536375 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.536350    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/602fb654-5a74-4edc-8044-e1077c3c558c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.544198 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.544177    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str5b\" (UniqueName: \"kubernetes.io/projected/602fb654-5a74-4edc-8044-e1077c3c558c-kube-api-access-str5b\") pod \"prometheus-k8s-0\" (UID: \"602fb654-5a74-4edc-8044-e1077c3c558c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.686831 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.686804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:21.856324 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:21.856300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:21.858238 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:40:21.858212    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602fb654_5a74_4edc_8044_e1077c3c558c.slice/crio-2b5b7f2e1b5750332f3b38d36d1d9d4a5e9193a4f2c77fb6b5bffb920a51235f WatchSource:0}: Error finding container 2b5b7f2e1b5750332f3b38d36d1d9d4a5e9193a4f2c77fb6b5bffb920a51235f: Status 404 returned error can't find the container with id 2b5b7f2e1b5750332f3b38d36d1d9d4a5e9193a4f2c77fb6b5bffb920a51235f
Jul 01 18:40:22.306073 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:22.306041    2583 generic.go:356] "Generic (PLEG): container finished" podID="602fb654-5a74-4edc-8044-e1077c3c558c" containerID="626c2d3cdef81dc25569e5951c640740456274ec776b13a5026ba6a88089b74f" exitCode=0
Jul 01 18:40:22.306465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:22.306125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerDied","Data":"626c2d3cdef81dc25569e5951c640740456274ec776b13a5026ba6a88089b74f"}
Jul 01 18:40:22.306465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:22.306208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"2b5b7f2e1b5750332f3b38d36d1d9d4a5e9193a4f2c77fb6b5bffb920a51235f"}
Jul 01 18:40:22.367121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:22.367093    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067ff1bc-289a-4b98-8360-0f354c5af7f9" path="/var/lib/kubelet/pods/067ff1bc-289a-4b98-8360-0f354c5af7f9/volumes"
Jul 01 18:40:23.313327 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313268    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"9fe3de7b6271d84452c77d635fb1df1c786e631b3d9ece504a4b3d0f1dbf3301"}
Jul 01 18:40:23.313740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"fdb80f0ef45d90337673dfbe776897f92bad030fcec8e2e1b8cfbeff70cf0d38"}
Jul 01 18:40:23.313740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"e283dd28f4cc0b76d7e67d622979d2f75a588b5efd8486027ead3c136a12cda0"}
Jul 01 18:40:23.313740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"8435f0b54289f34b219de60740521d5ff83a653cce625d61ebf132364fdf0d49"}
Jul 01 18:40:23.313740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"cfbfc8499665826fc16369a093a0900455045f1012faff9e9ec80fb1e47a587e"}
Jul 01 18:40:23.313740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.313494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"602fb654-5a74-4edc-8044-e1077c3c558c","Type":"ContainerStarted","Data":"2f2ff897c73b1d285e8b913cbc0bf3c482e599bfa4d5fc887a78c3bef55ef20f"}
Jul 01 18:40:23.359703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:23.359653    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3439015420000002 podStartE2EDuration="2.359633068s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.731526ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:40:23.357466353 +0000 UTC m=+181.373045918" watchObservedRunningTime="2026-07-01 18:40:23.359633068 +0000 UTC m=+181.375212614"
Jul 01 18:40:26.687341 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:26.687308    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:36.570402 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:40:36.570371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:41:01.590228 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.590173    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64f4b456f5-jndhn" podUID="88384853-f81c-4b04-864f-ceebc608477d" containerName="console" containerID="cri-o://08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0" gracePeriod=15
Jul 01 18:41:01.832386 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.832367    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:41:01.918158 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918270 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918270 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918183    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918270 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918228    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpc5s\" (UniqueName: \"kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918566 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918485    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert\") pod \"88384853-f81c-4b04-864f-ceebc608477d\" (UID: \"88384853-f81c-4b04-864f-ceebc608477d\") "
Jul 01 18:41:01.918782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918649    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca" (OuterVolumeSpecName: "service-ca") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:01.918782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918761    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-service-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:01.918782 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918772    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config" (OuterVolumeSpecName: "console-config") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:01.919005 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918782    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:01.919005 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.918958    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:41:01.920469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.920442    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s" (OuterVolumeSpecName: "kube-api-access-cpc5s") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "kube-api-access-cpc5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:41:01.920762 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.920733    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:41:01.920762 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:01.920744    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88384853-f81c-4b04-864f-ceebc608477d" (UID: "88384853-f81c-4b04-864f-ceebc608477d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:41:02.019724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019702    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-oauth-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.019724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019721    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-console-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.019862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019729    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88384853-f81c-4b04-864f-ceebc608477d-console-serving-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.019862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019738    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-trusted-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.019862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019746    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpc5s\" (UniqueName: \"kubernetes.io/projected/88384853-f81c-4b04-864f-ceebc608477d-kube-api-access-cpc5s\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.019862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.019754    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88384853-f81c-4b04-864f-ceebc608477d-oauth-serving-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:41:02.466433 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.466402    2583 generic.go:356] "Generic (PLEG): container finished" podID="88384853-f81c-4b04-864f-ceebc608477d" containerID="08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0" exitCode=0
Jul 01 18:41:02.466583 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.466452    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f4b456f5-jndhn"
Jul 01 18:41:02.466583 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.466468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f4b456f5-jndhn" event={"ID":"88384853-f81c-4b04-864f-ceebc608477d","Type":"ContainerDied","Data":"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"}
Jul 01 18:41:02.466583 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.466506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f4b456f5-jndhn" event={"ID":"88384853-f81c-4b04-864f-ceebc608477d","Type":"ContainerDied","Data":"9e90342104763870b259d1b0e022bf49bec13152dbcf4c0e4846a244fbae2e5f"}
Jul 01 18:41:02.466583 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.466528    2583 scope.go:117] "RemoveContainer" containerID="08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"
Jul 01 18:41:02.474582 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.474567    2583 scope.go:117] "RemoveContainer" containerID="08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"
Jul 01 18:41:02.474837 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:41:02.474812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0\": container with ID starting with 08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0 not found: ID does not exist" containerID="08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"
Jul 01 18:41:02.474931 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.474840    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0"} err="failed to get container status \"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0\": rpc error: code = NotFound desc = could not find container \"08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0\": container with ID starting with 08c92c5cfc77f1130eaca74d130deec7ab8fad5b10ac99138c3c08ffeadce7d0 not found: ID does not exist"
Jul 01 18:41:02.486445 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.486255    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:41:02.487726 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:02.487707    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64f4b456f5-jndhn"]
Jul 01 18:41:04.367060 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:04.367010    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88384853-f81c-4b04-864f-ceebc608477d" path="/var/lib/kubelet/pods/88384853-f81c-4b04-864f-ceebc608477d/volumes"
Jul 01 18:41:21.687606 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:21.687570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:41:21.703727 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:21.703700    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:41:22.564423 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:41:22.564403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:42:22.310091 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:42:22.310061    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:42:22.322610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:42:22.322581    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 18:42:22.322754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:42:22.322691    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:45:37.493123 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.493091    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qhjgv"]
Jul 01 18:45:37.493608 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.493436    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88384853-f81c-4b04-864f-ceebc608477d" containerName="console"
Jul 01 18:45:37.493608 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.493448    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88384853-f81c-4b04-864f-ceebc608477d" containerName="console"
Jul 01 18:45:37.493608 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.493499    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88384853-f81c-4b04-864f-ceebc608477d" containerName="console"
Jul 01 18:45:37.496480 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.496461    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.498868 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.498840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 18:45:37.498868 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.498858    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xs95r\""
Jul 01 18:45:37.499036 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.498843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:45:37.499775 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.499759    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:45:37.508865 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.508846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qhjgv"]
Jul 01 18:45:37.562441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.562418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af7a97bd-880e-43ce-8e15-102d8f6df7a2-cert\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.562543 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.562463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggq8\" (UniqueName: \"kubernetes.io/projected/af7a97bd-880e-43ce-8e15-102d8f6df7a2-kube-api-access-mggq8\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.663103 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.663078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af7a97bd-880e-43ce-8e15-102d8f6df7a2-cert\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.663199 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.663120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggq8\" (UniqueName: \"kubernetes.io/projected/af7a97bd-880e-43ce-8e15-102d8f6df7a2-kube-api-access-mggq8\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.665319 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.665302    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af7a97bd-880e-43ce-8e15-102d8f6df7a2-cert\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.672474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.672456    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggq8\" (UniqueName: \"kubernetes.io/projected/af7a97bd-880e-43ce-8e15-102d8f6df7a2-kube-api-access-mggq8\") pod \"odh-model-controller-84b54b7f97-qhjgv\" (UID: \"af7a97bd-880e-43ce-8e15-102d8f6df7a2\") " pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.805872 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.805811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:37.936950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.936927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qhjgv"]
Jul 01 18:45:37.942510 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:37.942486    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:45:38.531222 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:38.531184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qhjgv" event={"ID":"af7a97bd-880e-43ce-8e15-102d8f6df7a2","Type":"ContainerStarted","Data":"0731fba978a8c6aaa8ca59400820d808c6677fbbc41c53cb97ea001c2db8e13c"}
Jul 01 18:45:41.547261 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:41.547226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qhjgv" event={"ID":"af7a97bd-880e-43ce-8e15-102d8f6df7a2","Type":"ContainerStarted","Data":"579c1e4588a7b95d6e4c6ffd5a272a6fb8bb7d99f5f4ef6f6cffed63eb5ef7c1"}
Jul 01 18:45:41.547726 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:41.547353    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:41.571874 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:41.571826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-qhjgv" podStartSLOduration=1.6059736949999999 podStartE2EDuration="4.571811853s" totalImagesPullingTime="2.965838158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:45:41.569888008 +0000 UTC m=+499.585467553" watchObservedRunningTime="2026-07-01 18:45:41.571811853 +0000 UTC m=+499.587391396"
Jul 01 18:45:52.553175 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:52.553149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-qhjgv"
Jul 01 18:45:53.400115 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.400081    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9sg8p"]
Jul 01 18:45:53.402446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.402427    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.405386 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.405360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 18:45:53.405493 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.405390    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f5fqh\""
Jul 01 18:45:53.412555 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.412535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9sg8p"]
Jul 01 18:45:53.583055 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.583029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.583400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.583064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2p5f\" (UniqueName: \"kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.683499 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.683432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.683499 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.683469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2p5f\" (UniqueName: \"kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.683757 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.683739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.696263 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.696216    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2p5f\" (UniqueName: \"kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f\") pod \"s3-init-9sg8p\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") " pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.729802 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.729602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9sg8p"
Jul 01 18:45:53.860434 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:53.860412    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9sg8p"]
Jul 01 18:45:53.864679 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:45:53.864645    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197ec711_2289_4602_bda4_db0875c6bdcb.slice/crio-be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f WatchSource:0}: Error finding container be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f: Status 404 returned error can't find the container with id be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f
Jul 01 18:45:54.593679 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:54.593643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerStarted","Data":"be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f"}
Jul 01 18:45:59.618437 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:45:59.618381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerStarted","Data":"9bfa75574bf783d508d89d93a1b435d856b194c6e046d6ac7eca1bad603e2ab4"}
Jul 01 18:46:00.623652 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:46:00.623628    2583 generic.go:356] "Generic (PLEG): container finished" podID="197ec711-2289-4602-bda4-db0875c6bdcb" containerID="9bfa75574bf783d508d89d93a1b435d856b194c6e046d6ac7eca1bad603e2ab4" exitCode=0
Jul 01 18:46:00.623944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:46:00.623695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerDied","Data":"9bfa75574bf783d508d89d93a1b435d856b194c6e046d6ac7eca1bad603e2ab4"}
Jul 01 18:46:06.653043 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:46:06.653006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerStarted","Data":"5eeaf8f3279d2edbe0623ee4c502991f3b6597842ed277611e1a0ac17d3febcc"}
Jul 01 18:46:06.677101 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:46:06.677053    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9sg8p" podStartSLOduration=1.933634769 podStartE2EDuration="13.677036039s" totalImagesPullingTime="10.089617521s" totalInitContainerRuntime="1.653783749s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:46:06.676373039 +0000 UTC m=+524.691952582" watchObservedRunningTime="2026-07-01 18:46:06.677036039 +0000 UTC m=+524.692615583"
Jul 01 18:47:10.901160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:10.901129    2583 generic.go:356] "Generic (PLEG): container finished" podID="197ec711-2289-4602-bda4-db0875c6bdcb" containerID="5eeaf8f3279d2edbe0623ee4c502991f3b6597842ed277611e1a0ac17d3febcc" exitCode=0
Jul 01 18:47:10.901489 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:10.901202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerDied","Data":"5eeaf8f3279d2edbe0623ee4c502991f3b6597842ed277611e1a0ac17d3febcc"}
Jul 01 18:47:12.029171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.029149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9sg8p"
Jul 01 18:47:12.139101 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.139074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2p5f\" (UniqueName: \"kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f\") pod \"197ec711-2289-4602-bda4-db0875c6bdcb\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") "
Jul 01 18:47:12.139221 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.139132    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache\") pod \"197ec711-2289-4602-bda4-db0875c6bdcb\" (UID: \"197ec711-2289-4602-bda4-db0875c6bdcb\") "
Jul 01 18:47:12.141150 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.141122    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f" (OuterVolumeSpecName: "kube-api-access-b2p5f") pod "197ec711-2289-4602-bda4-db0875c6bdcb" (UID: "197ec711-2289-4602-bda4-db0875c6bdcb"). InnerVolumeSpecName "kube-api-access-b2p5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:47:12.158421 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.158372    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "197ec711-2289-4602-bda4-db0875c6bdcb" (UID: "197ec711-2289-4602-bda4-db0875c6bdcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:47:12.239788 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.239769    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2p5f\" (UniqueName: \"kubernetes.io/projected/197ec711-2289-4602-bda4-db0875c6bdcb-kube-api-access-b2p5f\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:47:12.239788 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.239787    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197ec711-2289-4602-bda4-db0875c6bdcb-model-cache\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:47:12.909965 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.909938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9sg8p" event={"ID":"197ec711-2289-4602-bda4-db0875c6bdcb","Type":"ContainerDied","Data":"be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f"}
Jul 01 18:47:12.909965 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.909956    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9sg8p"
Jul 01 18:47:12.910126 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:12.909973    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be11dd4e92027ea8a0ab5f09e1d037c1a46d32b86bb72cb24e134b41658d1a0f"
Jul 01 18:47:22.345742 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:22.345710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:47:22.347735 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:22.347697    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:47:24.580544 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580515    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:47:24.580902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580809    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197ec711-2289-4602-bda4-db0875c6bdcb" containerName="s3-init"
Jul 01 18:47:24.580902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580819    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="197ec711-2289-4602-bda4-db0875c6bdcb" containerName="s3-init"
Jul 01 18:47:24.580902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580838    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197ec711-2289-4602-bda4-db0875c6bdcb" containerName="download-hf-model"
Jul 01 18:47:24.580902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580843    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="197ec711-2289-4602-bda4-db0875c6bdcb" containerName="download-hf-model"
Jul 01 18:47:24.580902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.580894    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="197ec711-2289-4602-bda4-db0875c6bdcb" containerName="s3-init"
Jul 01 18:47:24.583943 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.583927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.586870 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.586840    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9995e-kube-rbac-proxy-sar-config\""
Jul 01 18:47:24.587000 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.586924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:47:24.587000 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.586929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:47:24.587761 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.587745    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9995e-predictor-serving-cert\""
Jul 01 18:47:24.587862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.587764    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-64f2j\""
Jul 01 18:47:24.587862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.587834    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:47:24.587862 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.587852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:47:24.596006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.595984    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:47:24.728512 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.728490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmtg\" (UniqueName: \"kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.728610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.728525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.728610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.728553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.728610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.728593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.728720 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.728618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829187 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829303 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmtg\" (UniqueName: \"kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829303 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829303 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829462 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829627    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.829933 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829915    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.830003 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.829957    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.831575 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.831538    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.839504 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.839480    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmtg\" (UniqueName: \"kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg\") pod \"isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:24.894326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:24.894308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:25.031826 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:25.031803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:47:25.034026 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:47:25.034000    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89f243b_f0ae_437b_9570_e0e9122218f9.slice/crio-9c860c6a90ec1c945b25f69f799b3686afc651b1d8df04eb07990325286a52a8 WatchSource:0}: Error finding container 9c860c6a90ec1c945b25f69f799b3686afc651b1d8df04eb07990325286a52a8: Status 404 returned error can't find the container with id 9c860c6a90ec1c945b25f69f799b3686afc651b1d8df04eb07990325286a52a8
Jul 01 18:47:25.965627 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:25.965590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerStarted","Data":"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"}
Jul 01 18:47:25.966085 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:25.965654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerStarted","Data":"9c860c6a90ec1c945b25f69f799b3686afc651b1d8df04eb07990325286a52a8"}
Jul 01 18:47:28.979120 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:28.979053    2583 generic.go:356] "Generic (PLEG): container finished" podID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerID="4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a" exitCode=0
Jul 01 18:47:28.979478 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:28.979120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerDied","Data":"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"}
Jul 01 18:47:44.062483 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:44.062450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerStarted","Data":"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"}
Jul 01 18:47:47.083710 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:47.083674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerStarted","Data":"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"}
Jul 01 18:47:50.099369 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:50.099338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerStarted","Data":"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"}
Jul 01 18:47:50.099749 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:50.099538    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:51.103581 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.103544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:51.104020 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.103621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:51.104779 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.104751    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:47:51.105445 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.105417    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:47:51.109166 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.109148    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:47:51.136200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:51.136158    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podStartSLOduration=3.379463585 podStartE2EDuration="27.136141629s" totalImagesPullingTime="20.16638715s" totalInitContainerRuntime="3.590290894s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:47:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:47:50.124638855 +0000 UTC m=+628.140218442" watchObservedRunningTime="2026-07-01 18:47:51.136141629 +0000 UTC m=+629.151721174"
Jul 01 18:47:52.107790 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:52.107750    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:47:52.108271 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:52.108007    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:47:53.111092 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:53.111053    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:47:53.111522 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:47:53.111423    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:03.111501 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:03.111458    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:03.111959 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:03.111921    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:13.111712 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:13.111669    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:13.112160 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:13.112070    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:23.111425 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:23.111382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:23.111930 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:23.111852    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:33.111567 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:33.111517    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:33.112051 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:33.111909    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:43.111611 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:43.111574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:43.112109 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:43.112085    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:53.111833 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:53.111784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:48:53.112355 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:48:53.112247    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:03.112559 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:03.112524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:03.113098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:03.112626    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:09.738212 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.738165    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:49:09.738939 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.738486    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" containerID="cri-o://d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959" gracePeriod=30
Jul 01 18:49:09.738939 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.738693    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" containerID="cri-o://19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915" gracePeriod=30
Jul 01 18:49:09.738939 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.738894    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" containerID="cri-o://9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78" gracePeriod=30
Jul 01 18:49:09.757071 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.757047    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:49:09.760871 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.760849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.763441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.763416    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-97f72-predictor-serving-cert\""
Jul 01 18:49:09.763542 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.763417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-97f72-kube-rbac-proxy-sar-config\""
Jul 01 18:49:09.775450 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.775427    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:49:09.819578 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.819553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.819675 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.819583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.819675 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.819606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.819675 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.819625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswbm\" (UniqueName: \"kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.819806 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.819726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.837966 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.837946    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:49:09.841236 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.841220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.845008 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.844990    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-97f72-kube-rbac-proxy-sar-config\""
Jul 01 18:49:09.845129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.845112    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-97f72-predictor-serving-cert\""
Jul 01 18:49:09.856983 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.856965    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:49:09.920648 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.920765 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.920765 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.920765 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.920950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswbm\" (UniqueName: \"kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.920950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wnq\" (UniqueName: \"kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.920950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.920950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.920950 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.921150 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.920962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:09.921275 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.921251    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.921462 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.921443    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.921509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.921481    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.923265 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.923245    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:09.933116 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:09.933094    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswbm\" (UniqueName: \"kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm\") pod \"isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:10.021806 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.021750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wnq\" (UniqueName: \"kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.021918 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.021803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.021918 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.021834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.021918 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.021861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.021918 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.021897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.022136 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:10.022112    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-serving-cert: secret "isvc-xgboost-graph-raw-97f72-predictor-serving-cert" not found
Jul 01 18:49:10.022194 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:10.022188    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls podName:4e6182d1-8191-45e8-bee8-ce37ec945e61 nodeName:}" failed. No retries permitted until 2026-07-01 18:49:10.522169627 +0000 UTC m=+708.537749155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls") pod "isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61") : secret "isvc-xgboost-graph-raw-97f72-predictor-serving-cert" not found
Jul 01 18:49:10.022274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.022252    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.022444 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.022427    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.022495 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.022485    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.032509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.032489    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wnq\" (UniqueName: \"kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.072397 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.072371    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:10.204317 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.204276    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:49:10.206472 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:49:10.206443    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c5b5f9_c502_491d_a1c9_cbd415b2b0fe.slice/crio-18d54b392b8d8445732d12a20374867cbf5444ad009510e3f436d39801ba8a37 WatchSource:0}: Error finding container 18d54b392b8d8445732d12a20374867cbf5444ad009510e3f436d39801ba8a37: Status 404 returned error can't find the container with id 18d54b392b8d8445732d12a20374867cbf5444ad009510e3f436d39801ba8a37
Jul 01 18:49:10.425503 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.425477    2583 generic.go:356] "Generic (PLEG): container finished" podID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerID="19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915" exitCode=2
Jul 01 18:49:10.425660 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.425552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerDied","Data":"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"}
Jul 01 18:49:10.427062 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.427044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerStarted","Data":"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"}
Jul 01 18:49:10.427166 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.427084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerStarted","Data":"18d54b392b8d8445732d12a20374867cbf5444ad009510e3f436d39801ba8a37"}
Jul 01 18:49:10.526770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.526744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.529244 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.529220    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") pod \"isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.751220 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.751152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:10.879101 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:10.879079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:49:10.881308 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:49:10.881258    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6182d1_8191_45e8_bee8_ce37ec945e61.slice/crio-2746d8c00b2a866ea7a2f85c12df853e8efc86a1cd6297823d7463636fbc09d9 WatchSource:0}: Error finding container 2746d8c00b2a866ea7a2f85c12df853e8efc86a1cd6297823d7463636fbc09d9: Status 404 returned error can't find the container with id 2746d8c00b2a866ea7a2f85c12df853e8efc86a1cd6297823d7463636fbc09d9
Jul 01 18:49:11.104118 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:11.104020    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:11.432754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:11.432720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerStarted","Data":"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"}
Jul 01 18:49:11.432886 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:11.432799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerStarted","Data":"2746d8c00b2a866ea7a2f85c12df853e8efc86a1cd6297823d7463636fbc09d9"}
Jul 01 18:49:13.111961 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:13.111913    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:49:13.112348 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:13.112221    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:14.446540 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:14.446513    2583 generic.go:356] "Generic (PLEG): container finished" podID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerID="d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959" exitCode=0
Jul 01 18:49:14.446907 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:14.446579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerDied","Data":"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"}
Jul 01 18:49:14.448006 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:14.447987    2583 generic.go:356] "Generic (PLEG): container finished" podID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerID="9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f" exitCode=0
Jul 01 18:49:14.448121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:14.448016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerDied","Data":"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"}
Jul 01 18:49:15.453743 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.453666    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerID="e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4" exitCode=0
Jul 01 18:49:15.454213 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.453737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerDied","Data":"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"}
Jul 01 18:49:15.456175 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.456151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerStarted","Data":"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"}
Jul 01 18:49:15.456261 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.456198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerStarted","Data":"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"}
Jul 01 18:49:15.456401 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.456387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:15.494944 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:15.494905    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podStartSLOduration=2.794252762 podStartE2EDuration="6.494892639s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700639877s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:49:15.493388409 +0000 UTC m=+713.508967977" watchObservedRunningTime="2026-07-01 18:49:15.494892639 +0000 UTC m=+713.510472183"
Jul 01 18:49:16.104252 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:16.104210    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:16.462246 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:16.462191    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:16.463702 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:16.463674    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:17.468774 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:17.468394    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:21.104350 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:21.104305    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:21.104789 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:21.104456    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:22.474429 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:22.474404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:49:22.475119 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:22.475089    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:23.111198 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:23.111132    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:49:23.111595 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:23.111564    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:26.104373 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:26.104328    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:31.104573 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:31.104529    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:32.475194 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:32.475159    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:33.112011 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:33.111971    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 18:49:33.112207 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:33.112120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:33.112390 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:33.112347    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:33.112516 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:33.112502    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:36.104163 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:36.104112    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 18:49:36.560208 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:36.560180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerStarted","Data":"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"}
Jul 01 18:49:37.566428 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:37.566399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerStarted","Data":"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"}
Jul 01 18:49:37.566776 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:37.566508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:37.588009 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:37.587970    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podStartSLOduration=3.857135548 podStartE2EDuration="28.587956466s" totalImagesPullingTime="20.999447522s" totalInitContainerRuntime="3.731373396s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:49:37.58720054 +0000 UTC m=+735.602780129" watchObservedRunningTime="2026-07-01 18:49:37.587956466 +0000 UTC m=+735.603536010"
Jul 01 18:49:38.570727 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:38.570689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:38.571771 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:38.571748    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:49:39.574665 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:39.574625    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:49:39.893977 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:39.893958    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:40.084426 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls\") pod \"e89f243b-f0ae-437b-9570-e0e9122218f9\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") "
Jul 01 18:49:40.084426 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084427    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config\") pod \"e89f243b-f0ae-437b-9570-e0e9122218f9\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") "
Jul 01 18:49:40.084654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert\") pod \"e89f243b-f0ae-437b-9570-e0e9122218f9\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") "
Jul 01 18:49:40.084654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location\") pod \"e89f243b-f0ae-437b-9570-e0e9122218f9\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") "
Jul 01 18:49:40.084654 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmtg\" (UniqueName: \"kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg\") pod \"e89f243b-f0ae-437b-9570-e0e9122218f9\" (UID: \"e89f243b-f0ae-437b-9570-e0e9122218f9\") "
Jul 01 18:49:40.084843 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084819    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e89f243b-f0ae-437b-9570-e0e9122218f9" (UID: "e89f243b-f0ae-437b-9570-e0e9122218f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:49:40.084895 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084840    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e89f243b-f0ae-437b-9570-e0e9122218f9" (UID: "e89f243b-f0ae-437b-9570-e0e9122218f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:49:40.084895 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.084817    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e89f243b-f0ae-437b-9570-e0e9122218f9" (UID: "e89f243b-f0ae-437b-9570-e0e9122218f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:49:40.086566 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.086545    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg" (OuterVolumeSpecName: "kube-api-access-rtmtg") pod "e89f243b-f0ae-437b-9570-e0e9122218f9" (UID: "e89f243b-f0ae-437b-9570-e0e9122218f9"). InnerVolumeSpecName "kube-api-access-rtmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:49:40.086645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.086627    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e89f243b-f0ae-437b-9570-e0e9122218f9" (UID: "e89f243b-f0ae-437b-9570-e0e9122218f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:49:40.185143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.185088    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmtg\" (UniqueName: \"kubernetes.io/projected/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-api-access-rtmtg\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:49:40.185143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.185109    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89f243b-f0ae-437b-9570-e0e9122218f9-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:49:40.185143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.185119    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:49:40.185143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.185127    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e89f243b-f0ae-437b-9570-e0e9122218f9-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:49:40.185143 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.185135    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e89f243b-f0ae-437b-9570-e0e9122218f9-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:49:40.580171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.580090    2583 generic.go:356] "Generic (PLEG): container finished" podID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerID="9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78" exitCode=0
Jul 01 18:49:40.580171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.580149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"
Jul 01 18:49:40.580618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.580174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerDied","Data":"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"}
Jul 01 18:49:40.580618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.580246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc" event={"ID":"e89f243b-f0ae-437b-9570-e0e9122218f9","Type":"ContainerDied","Data":"9c860c6a90ec1c945b25f69f799b3686afc651b1d8df04eb07990325286a52a8"}
Jul 01 18:49:40.580618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.580278    2583 scope.go:117] "RemoveContainer" containerID="9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"
Jul 01 18:49:40.590630 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.590589    2583 scope.go:117] "RemoveContainer" containerID="19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"
Jul 01 18:49:40.598509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.598493    2583 scope.go:117] "RemoveContainer" containerID="d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"
Jul 01 18:49:40.603069 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.603043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:49:40.605853 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.605839    2583 scope.go:117] "RemoveContainer" containerID="4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"
Jul 01 18:49:40.610408 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.610386    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9995e-predictor-57944bc586-762nc"]
Jul 01 18:49:40.613564 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.613548    2583 scope.go:117] "RemoveContainer" containerID="9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"
Jul 01 18:49:40.613813 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:40.613796    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78\": container with ID starting with 9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78 not found: ID does not exist" containerID="9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"
Jul 01 18:49:40.613860 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.613820    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78"} err="failed to get container status \"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78\": rpc error: code = NotFound desc = could not find container \"9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78\": container with ID starting with 9a0498c64baa734bc5ab05cc1e7c791547ee46e2eaf0c5130aaa9377599aaa78 not found: ID does not exist"
Jul 01 18:49:40.613860 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.613835    2583 scope.go:117] "RemoveContainer" containerID="19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"
Jul 01 18:49:40.614076 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:40.614061    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915\": container with ID starting with 19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915 not found: ID does not exist" containerID="19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"
Jul 01 18:49:40.614124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.614080    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915"} err="failed to get container status \"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915\": rpc error: code = NotFound desc = could not find container \"19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915\": container with ID starting with 19273a467133a142d31e1aedd186150f5d883f40d89d3da0ead5f9ca34b61915 not found: ID does not exist"
Jul 01 18:49:40.614124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.614097    2583 scope.go:117] "RemoveContainer" containerID="d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"
Jul 01 18:49:40.614279 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:40.614263    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959\": container with ID starting with d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959 not found: ID does not exist" containerID="d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"
Jul 01 18:49:40.614333 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.614298    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959"} err="failed to get container status \"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959\": rpc error: code = NotFound desc = could not find container \"d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959\": container with ID starting with d2e7d3c510c9f466a17a7a62e7d09db8572edf6f8be4df1bcc6c2dc25c70c959 not found: ID does not exist"
Jul 01 18:49:40.614333 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.614311    2583 scope.go:117] "RemoveContainer" containerID="4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"
Jul 01 18:49:40.614489 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:49:40.614474    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a\": container with ID starting with 4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a not found: ID does not exist" containerID="4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"
Jul 01 18:49:40.614530 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:40.614492    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a"} err="failed to get container status \"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a\": rpc error: code = NotFound desc = could not find container \"4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a\": container with ID starting with 4e6d7caf510256d1247bf31e622c3af089a758aa8ce8ec4eadcf4ad84cc7b10a not found: ID does not exist"
Jul 01 18:49:42.366794 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:42.366759    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" path="/var/lib/kubelet/pods/e89f243b-f0ae-437b-9570-e0e9122218f9/volumes"
Jul 01 18:49:42.475028 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:42.474989    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:44.580434 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:44.580410    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:49:44.580976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:44.580947    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:49:52.475242 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:52.475205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:49:54.581040 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:49:54.580985    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:50:02.475272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:02.475237    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:50:04.581253 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:04.581218    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:50:12.474971 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:12.474934    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:50:14.581752 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:14.581709    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:50:22.475714 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:22.475683    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:50:24.580968 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:24.580926    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 18:50:32.475423 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:32.475395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:50:34.581505 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:50:34.581479    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:51:00.110609 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.108408    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:51:00.110609 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.108801    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" containerID="cri-o://757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3" gracePeriod=30
Jul 01 18:51:00.110609 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.109044    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kube-rbac-proxy" containerID="cri-o://94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40" gracePeriod=30
Jul 01 18:51:00.119225 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.118810    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:51:00.119225 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119184    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy"
Jul 01 18:51:00.119225 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119201    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy"
Jul 01 18:51:00.119225 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119225    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119234    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119245    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119254    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119277    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="storage-initializer"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119304    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="storage-initializer"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119398    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kube-rbac-proxy"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119413    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="kserve-container"
Jul 01 18:51:00.119803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.119421    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89f243b-f0ae-437b-9570-e0e9122218f9" containerName="agent"
Jul 01 18:51:00.121559 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.121544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.124147 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.124124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e8497-predictor-serving-cert\""
Jul 01 18:51:00.124607 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.124588    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e8497-kube-rbac-proxy-sar-config\""
Jul 01 18:51:00.134250 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.134225    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:51:00.204421 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.204392    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:51:00.206846 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.206828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.209784 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.209397    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e8497-kube-rbac-proxy-sar-config\""
Jul 01 18:51:00.209784 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.209590    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e8497-predictor-serving-cert\""
Jul 01 18:51:00.227582 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.227558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:51:00.257530 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.257504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.257637 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.257546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.257637 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.257569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlq8v\" (UniqueName: \"kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.257637 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.257617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.257766 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.257678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.266504 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.266481    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:51:00.266712 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.266678    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container" containerID="cri-o://6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d" gracePeriod=30
Jul 01 18:51:00.266827 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.266721    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kube-rbac-proxy" containerID="cri-o://86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac" gracePeriod=30
Jul 01 18:51:00.358849 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.358819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlq8v\" (UniqueName: \"kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.358964 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.358854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.358964 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.358881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.358964 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.358938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.359129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.358981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.359129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf5j\" (UniqueName: \"kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.359129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.359129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.359129 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.359460 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.359506 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359462    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.359656 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.359724 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.359709    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.361521 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.361452    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.371440 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.371417    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlq8v\" (UniqueName: \"kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v\") pod \"isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.438276 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.438251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:00.459812 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.459789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf5j\" (UniqueName: \"kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.459920 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.459823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.459920 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.459853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.460041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.459935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.460041 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:00.459957    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-e8497-predictor-serving-cert" not found
Jul 01 18:51:00.460041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.459973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.460041 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:00.460031    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls podName:ad26e5f8-d3c4-4b08-b958-006b7270464b nodeName:}" failed. No retries permitted until 2026-07-01 18:51:00.960010012 +0000 UTC m=+818.975589548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b") : secret "isvc-xgboost-graph-raw-hpa-e8497-predictor-serving-cert" not found
Jul 01 18:51:00.460258 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.460192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.460590 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.460570    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.460668 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.460651    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.474617 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.474599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf5j\" (UniqueName: \"kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.572012 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.571988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:51:00.574071 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:51:00.574044    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47711f2_01a1_44b2_9768_7d338c8f9485.slice/crio-bef568db7861c8a982831c07e5b107ba9be3f864b245b0726a7ce6ad414b9d9b WatchSource:0}: Error finding container bef568db7861c8a982831c07e5b107ba9be3f864b245b0726a7ce6ad414b9d9b: Status 404 returned error can't find the container with id bef568db7861c8a982831c07e5b107ba9be3f864b245b0726a7ce6ad414b9d9b
Jul 01 18:51:00.912423 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.912395    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerID="86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac" exitCode=2
Jul 01 18:51:00.912601 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.912463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerDied","Data":"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"}
Jul 01 18:51:00.914507 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.914486    2583 generic.go:356] "Generic (PLEG): container finished" podID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerID="94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40" exitCode=2
Jul 01 18:51:00.914616 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.914549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerDied","Data":"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"}
Jul 01 18:51:00.915941 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.915913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerStarted","Data":"6ae133a354223b397516e50a64bf1b85692a85b86a5d12af44108ac6e717ed77"}
Jul 01 18:51:00.916050 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.915968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerStarted","Data":"bef568db7861c8a982831c07e5b107ba9be3f864b245b0726a7ce6ad414b9d9b"}
Jul 01 18:51:00.966023 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.965991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:00.968301 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:00.968270    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:01.118948 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:01.118913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:01.251013 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:01.250773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:51:01.254202 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:51:01.254170    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad26e5f8_d3c4_4b08_b958_006b7270464b.slice/crio-abca17e4876b1b2c3f0e0e1463462f0c06a419a794c10d96e1f4194b484b2218 WatchSource:0}: Error finding container abca17e4876b1b2c3f0e0e1463462f0c06a419a794c10d96e1f4194b484b2218: Status 404 returned error can't find the container with id abca17e4876b1b2c3f0e0e1463462f0c06a419a794c10d96e1f4194b484b2218
Jul 01 18:51:01.921577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:01.921543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerStarted","Data":"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"}
Jul 01 18:51:01.921756 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:01.921600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerStarted","Data":"abca17e4876b1b2c3f0e0e1463462f0c06a419a794c10d96e1f4194b484b2218"}
Jul 01 18:51:02.470200 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:02.470161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 01 18:51:02.475501 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:02.475477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 18:51:03.612789 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.612767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:51:03.789024 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.788949    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert\") pod \"4e6182d1-8191-45e8-bee8-ce37ec945e61\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") "
Jul 01 18:51:03.789024 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.788996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config\") pod \"4e6182d1-8191-45e8-bee8-ce37ec945e61\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") "
Jul 01 18:51:03.789224 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") pod \"4e6182d1-8191-45e8-bee8-ce37ec945e61\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") "
Jul 01 18:51:03.789224 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789174    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65wnq\" (UniqueName: \"kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq\") pod \"4e6182d1-8191-45e8-bee8-ce37ec945e61\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") "
Jul 01 18:51:03.789224 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location\") pod \"4e6182d1-8191-45e8-bee8-ce37ec945e61\" (UID: \"4e6182d1-8191-45e8-bee8-ce37ec945e61\") "
Jul 01 18:51:03.789417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789374    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e6182d1-8191-45e8-bee8-ce37ec945e61" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:51:03.789417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789383    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e6182d1-8191-45e8-bee8-ce37ec945e61" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:51:03.789590 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789567    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e6182d1-8191-45e8-bee8-ce37ec945e61" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:51:03.789671 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789634    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:03.789671 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.789654    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:03.791205 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.791187    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e6182d1-8191-45e8-bee8-ce37ec945e61" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:51:03.791326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.791309    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq" (OuterVolumeSpecName: "kube-api-access-65wnq") pod "4e6182d1-8191-45e8-bee8-ce37ec945e61" (UID: "4e6182d1-8191-45e8-bee8-ce37ec945e61"). InnerVolumeSpecName "kube-api-access-65wnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:51:03.890731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.890702    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6182d1-8191-45e8-bee8-ce37ec945e61-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:03.890731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.890729    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wnq\" (UniqueName: \"kubernetes.io/projected/4e6182d1-8191-45e8-bee8-ce37ec945e61-kube-api-access-65wnq\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:03.890916 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.890743    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6182d1-8191-45e8-bee8-ce37ec945e61-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:03.931374 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.931344    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerID="6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d" exitCode=0
Jul 01 18:51:03.931474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.931391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerDied","Data":"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"}
Jul 01 18:51:03.931474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.931407    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"
Jul 01 18:51:03.931474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.931466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69" event={"ID":"4e6182d1-8191-45e8-bee8-ce37ec945e61","Type":"ContainerDied","Data":"2746d8c00b2a866ea7a2f85c12df853e8efc86a1cd6297823d7463636fbc09d9"}
Jul 01 18:51:03.931579 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.931493    2583 scope.go:117] "RemoveContainer" containerID="86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"
Jul 01 18:51:03.940570 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.940550    2583 scope.go:117] "RemoveContainer" containerID="6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"
Jul 01 18:51:03.948078 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.948058    2583 scope.go:117] "RemoveContainer" containerID="e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"
Jul 01 18:51:03.955145 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.955124    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:51:03.955564 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.955542    2583 scope.go:117] "RemoveContainer" containerID="86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"
Jul 01 18:51:03.955830 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:03.955811    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac\": container with ID starting with 86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac not found: ID does not exist" containerID="86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"
Jul 01 18:51:03.955890 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.955846    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac"} err="failed to get container status \"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac\": rpc error: code = NotFound desc = could not find container \"86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac\": container with ID starting with 86902eb93f3791d5448c6ccf736ceb0ba18e071600c0c43a967c3fc18bd62aac not found: ID does not exist"
Jul 01 18:51:03.955890 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.955863    2583 scope.go:117] "RemoveContainer" containerID="6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"
Jul 01 18:51:03.956099 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:03.956082    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d\": container with ID starting with 6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d not found: ID does not exist" containerID="6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"
Jul 01 18:51:03.956164 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.956108    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d"} err="failed to get container status \"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d\": rpc error: code = NotFound desc = could not find container \"6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d\": container with ID starting with 6e374d7545dd8343bc22e1a900aa90b66ecbf0e8b7b9a66cae86834d56a3670d not found: ID does not exist"
Jul 01 18:51:03.956164 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.956131    2583 scope.go:117] "RemoveContainer" containerID="e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"
Jul 01 18:51:03.956390 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:03.956374    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4\": container with ID starting with e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4 not found: ID does not exist" containerID="e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"
Jul 01 18:51:03.956438 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.956396    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4"} err="failed to get container status \"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4\": rpc error: code = NotFound desc = could not find container \"e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4\": container with ID starting with e2859659f542823f73e70bfe6aaca5fd5b87a201feed5ce02c7d31187cb449d4 not found: ID does not exist"
Jul 01 18:51:03.960954 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:03.960934    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-97f72-predictor-8db4f5cf7-tkv69"]
Jul 01 18:51:04.349955 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.349935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:51:04.366255 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.366226    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" path="/var/lib/kubelet/pods/4e6182d1-8191-45e8-bee8-ce37ec945e61/volumes"
Jul 01 18:51:04.495734 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.495712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location\") pod \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") "
Jul 01 18:51:04.495824 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.495782    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xswbm\" (UniqueName: \"kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm\") pod \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") "
Jul 01 18:51:04.495824 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.495802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert\") pod \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") "
Jul 01 18:51:04.495914 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.495822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls\") pod \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") "
Jul 01 18:51:04.495914 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.495846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config\") pod \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\" (UID: \"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe\") "
Jul 01 18:51:04.496069 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.496032    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" (UID: "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:51:04.496240 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.496221    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" (UID: "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:51:04.496353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.496333    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" (UID: "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:51:04.497886 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.497854    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm" (OuterVolumeSpecName: "kube-api-access-xswbm") pod "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" (UID: "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe"). InnerVolumeSpecName "kube-api-access-xswbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:51:04.498017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.497885    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" (UID: "63c5b5f9-c502-491d-a1c9-cbd415b2b0fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:51:04.596442 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.596420    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswbm\" (UniqueName: \"kubernetes.io/projected/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-api-access-xswbm\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:04.596442 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.596440    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:04.596577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.596449    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:04.596577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.596477    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:04.596577 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.596486    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:51:04.938557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.938532    2583 generic.go:356] "Generic (PLEG): container finished" podID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerID="757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3" exitCode=0
Jul 01 18:51:04.939098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.938587    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"
Jul 01 18:51:04.939098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.938601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerDied","Data":"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"}
Jul 01 18:51:04.939098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.938650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk" event={"ID":"63c5b5f9-c502-491d-a1c9-cbd415b2b0fe","Type":"ContainerDied","Data":"18d54b392b8d8445732d12a20374867cbf5444ad009510e3f436d39801ba8a37"}
Jul 01 18:51:04.939098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.938681    2583 scope.go:117] "RemoveContainer" containerID="94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"
Jul 01 18:51:04.940469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.940447    2583 generic.go:356] "Generic (PLEG): container finished" podID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerID="6ae133a354223b397516e50a64bf1b85692a85b86a5d12af44108ac6e717ed77" exitCode=0
Jul 01 18:51:04.940606 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.940515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerDied","Data":"6ae133a354223b397516e50a64bf1b85692a85b86a5d12af44108ac6e717ed77"}
Jul 01 18:51:04.947757 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.947737    2583 scope.go:117] "RemoveContainer" containerID="757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"
Jul 01 18:51:04.955721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.955698    2583 scope.go:117] "RemoveContainer" containerID="9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"
Jul 01 18:51:04.968568 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.968551    2583 scope.go:117] "RemoveContainer" containerID="94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"
Jul 01 18:51:04.968909 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:04.968884    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40\": container with ID starting with 94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40 not found: ID does not exist" containerID="94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"
Jul 01 18:51:04.968975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.968917    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40"} err="failed to get container status \"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40\": rpc error: code = NotFound desc = could not find container \"94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40\": container with ID starting with 94264af2cf7ea4bf3c20343844e9dd059936a1c9f98011ae558334a0126dbd40 not found: ID does not exist"
Jul 01 18:51:04.968975 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.968934    2583 scope.go:117] "RemoveContainer" containerID="757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"
Jul 01 18:51:04.969155 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:04.969139    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3\": container with ID starting with 757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3 not found: ID does not exist" containerID="757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"
Jul 01 18:51:04.969203 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.969159    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3"} err="failed to get container status \"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3\": rpc error: code = NotFound desc = could not find container \"757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3\": container with ID starting with 757402fdecf5a2ce0b76e50b08e1bae616d7c39ffe12cb6dffc4fadb03ba89c3 not found: ID does not exist"
Jul 01 18:51:04.969203 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.969173    2583 scope.go:117] "RemoveContainer" containerID="9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"
Jul 01 18:51:04.969441 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:51:04.969425    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f\": container with ID starting with 9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f not found: ID does not exist" containerID="9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"
Jul 01 18:51:04.969498 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.969446    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f"} err="failed to get container status \"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f\": rpc error: code = NotFound desc = could not find container \"9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f\": container with ID starting with 9fd3f409acfcddb611d7d013a4e85c7634d0e3865283a46dd0dc8571590fba8f not found: ID does not exist"
Jul 01 18:51:04.980730 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.980660    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:51:04.988211 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:04.988185    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-97f72-predictor-6d7f57c67d-c5mkk"]
Jul 01 18:51:05.948927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.948887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerStarted","Data":"80dd8a94a7f99514a4f1f698fa7a119b8b3fbf3555e07ecafdd2e0ba87a9c07e"}
Jul 01 18:51:05.949374 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.948959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerStarted","Data":"4eca312ed3400c2a4842d807bdbcf89d85549d95eee3c876943631fae7212274"}
Jul 01 18:51:05.949374 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.949162    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:05.950377 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.950357    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerID="a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90" exitCode=0
Jul 01 18:51:05.950482 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.950380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerDied","Data":"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"}
Jul 01 18:51:05.976402 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:05.976362    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podStartSLOduration=2.585060116 podStartE2EDuration="5.976347961s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.391287845s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:51:05.974950622 +0000 UTC m=+823.990530178" watchObservedRunningTime="2026-07-01 18:51:05.976347961 +0000 UTC m=+823.991927505"
Jul 01 18:51:06.366733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.366703    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" path="/var/lib/kubelet/pods/63c5b5f9-c502-491d-a1c9-cbd415b2b0fe/volumes"
Jul 01 18:51:06.956889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.956858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerStarted","Data":"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"}
Jul 01 18:51:06.957358 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.957134    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:06.957358 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.957171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerStarted","Data":"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"}
Jul 01 18:51:06.957486 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.957420    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:06.957581 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.957563    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:06.958118 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.958094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:06.958465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.958446    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:06.978927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:06.978885    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podStartSLOduration=3.294590787 podStartE2EDuration="6.978872499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684281712s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:51:06.977135991 +0000 UTC m=+824.992715571" watchObservedRunningTime="2026-07-01 18:51:06.978872499 +0000 UTC m=+824.994452043"
Jul 01 18:51:07.962146 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:07.962097    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:07.962613 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:07.962204    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:12.967280 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:12.967253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:51:12.967734 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:12.967459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:51:12.967886 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:12.967864    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:12.967945 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:12.967893    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:22.968075 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:22.968036    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:22.968477 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:22.968036    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:32.967972 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:32.967935    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:32.967972 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:32.967953    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:42.968710 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:42.968675    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:51:42.969076 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:42.968675    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:52.968358 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:52.968322    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:51:52.968723 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:51:52.968327    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:52:02.968468 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:02.968432    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:52:02.969098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:02.969081    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:52:12.968848 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:12.968810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:52:22.375549 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:22.375477    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:52:22.376017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:22.375613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:52:22.968461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:22.968428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:52:50.452031 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.452002    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:52:50.452551 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.452235    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" containerID="cri-o://4eca312ed3400c2a4842d807bdbcf89d85549d95eee3c876943631fae7212274" gracePeriod=30
Jul 01 18:52:50.452551 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.452268    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kube-rbac-proxy" containerID="cri-o://80dd8a94a7f99514a4f1f698fa7a119b8b3fbf3555e07ecafdd2e0ba87a9c07e" gracePeriod=30
Jul 01 18:52:50.509436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.509409    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:52:50.509671 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.509633    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" containerID="cri-o://283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c" gracePeriod=30
Jul 01 18:52:50.509723 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:50.509656    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kube-rbac-proxy" containerID="cri-o://1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c" gracePeriod=30
Jul 01 18:52:51.396889 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:51.396857    2583 generic.go:356] "Generic (PLEG): container finished" podID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerID="80dd8a94a7f99514a4f1f698fa7a119b8b3fbf3555e07ecafdd2e0ba87a9c07e" exitCode=2
Jul 01 18:52:51.397071 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:51.396928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerDied","Data":"80dd8a94a7f99514a4f1f698fa7a119b8b3fbf3555e07ecafdd2e0ba87a9c07e"}
Jul 01 18:52:51.398867 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:51.398841    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerID="1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c" exitCode=2
Jul 01 18:52:51.398993 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:51.398912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerDied","Data":"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"}
Jul 01 18:52:52.963223 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:52.963187    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 18:52:52.963646 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:52.963185    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 18:52:52.968528 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:52.968504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 18:52:52.968642 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:52.968502    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 18:52:53.749452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.749429    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:52:53.804415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config\") pod \"ad26e5f8-d3c4-4b08-b958-006b7270464b\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") "
Jul 01 18:52:53.804415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txf5j\" (UniqueName: \"kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j\") pod \"ad26e5f8-d3c4-4b08-b958-006b7270464b\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") "
Jul 01 18:52:53.804558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert\") pod \"ad26e5f8-d3c4-4b08-b958-006b7270464b\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") "
Jul 01 18:52:53.804558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804470    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location\") pod \"ad26e5f8-d3c4-4b08-b958-006b7270464b\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") "
Jul 01 18:52:53.804558 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") pod \"ad26e5f8-d3c4-4b08-b958-006b7270464b\" (UID: \"ad26e5f8-d3c4-4b08-b958-006b7270464b\") "
Jul 01 18:52:53.804822 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804796    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad26e5f8-d3c4-4b08-b958-006b7270464b" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:53.804903 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804799    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad26e5f8-d3c4-4b08-b958-006b7270464b" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:52:53.804903 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.804864    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad26e5f8-d3c4-4b08-b958-006b7270464b" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:53.806803 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.806784    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad26e5f8-d3c4-4b08-b958-006b7270464b" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:52:53.806871 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.806794    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j" (OuterVolumeSpecName: "kube-api-access-txf5j") pod "ad26e5f8-d3c4-4b08-b958-006b7270464b" (UID: "ad26e5f8-d3c4-4b08-b958-006b7270464b"). InnerVolumeSpecName "kube-api-access-txf5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:52:53.905163 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.905138    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:53.905163 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.905160    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad26e5f8-d3c4-4b08-b958-006b7270464b-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:53.905314 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.905170    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad26e5f8-d3c4-4b08-b958-006b7270464b-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:53.905314 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.905179    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:53.905314 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:53.905186    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txf5j\" (UniqueName: \"kubernetes.io/projected/ad26e5f8-d3c4-4b08-b958-006b7270464b-kube-api-access-txf5j\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:54.414519 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.414480    2583 generic.go:356] "Generic (PLEG): container finished" podID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerID="4eca312ed3400c2a4842d807bdbcf89d85549d95eee3c876943631fae7212274" exitCode=0
Jul 01 18:52:54.414926 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.414547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerDied","Data":"4eca312ed3400c2a4842d807bdbcf89d85549d95eee3c876943631fae7212274"}
Jul 01 18:52:54.416324 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.416300    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerID="283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c" exitCode=0
Jul 01 18:52:54.416441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.416346    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"
Jul 01 18:52:54.416441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.416383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerDied","Data":"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"}
Jul 01 18:52:54.416441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.416438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x" event={"ID":"ad26e5f8-d3c4-4b08-b958-006b7270464b","Type":"ContainerDied","Data":"abca17e4876b1b2c3f0e0e1463462f0c06a419a794c10d96e1f4194b484b2218"}
Jul 01 18:52:54.416610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.416466    2583 scope.go:117] "RemoveContainer" containerID="1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"
Jul 01 18:52:54.426268 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.426250    2583 scope.go:117] "RemoveContainer" containerID="283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"
Jul 01 18:52:54.434745 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.434730    2583 scope.go:117] "RemoveContainer" containerID="a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"
Jul 01 18:52:54.438646 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.438623    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:52:54.443270 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.443248    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e8497-predictor-7764487c4f-nrg2x"]
Jul 01 18:52:54.445260 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.445235    2583 scope.go:117] "RemoveContainer" containerID="1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"
Jul 01 18:52:54.445605 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:52:54.445583    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c\": container with ID starting with 1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c not found: ID does not exist" containerID="1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"
Jul 01 18:52:54.445683 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.445608    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c"} err="failed to get container status \"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c\": rpc error: code = NotFound desc = could not find container \"1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c\": container with ID starting with 1f7e27ea187295d9236322852eeefe42f04fcb27aa52e7d336c98a7c4fa1836c not found: ID does not exist"
Jul 01 18:52:54.445683 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.445625    2583 scope.go:117] "RemoveContainer" containerID="283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"
Jul 01 18:52:54.445882 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:52:54.445863    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c\": container with ID starting with 283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c not found: ID does not exist" containerID="283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"
Jul 01 18:52:54.445961 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.445900    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c"} err="failed to get container status \"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c\": rpc error: code = NotFound desc = could not find container \"283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c\": container with ID starting with 283399d3e1e48b4bb2abe2bc5f5fb8db67656f28c19e44c87a608685be87451c not found: ID does not exist"
Jul 01 18:52:54.445961 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.445922    2583 scope.go:117] "RemoveContainer" containerID="a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"
Jul 01 18:52:54.446194 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:52:54.446176    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90\": container with ID starting with a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90 not found: ID does not exist" containerID="a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"
Jul 01 18:52:54.446236 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.446199    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90"} err="failed to get container status \"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90\": rpc error: code = NotFound desc = could not find container \"a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90\": container with ID starting with a5e419e2a0842fabcc45974bd3d5319b99e719ac8dd996be238753420af64b90 not found: ID does not exist"
Jul 01 18:52:54.493330 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.493312    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:52:54.508897 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.508875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location\") pod \"c47711f2-01a1-44b2-9768-7d338c8f9485\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") "
Jul 01 18:52:54.508986 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.508904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config\") pod \"c47711f2-01a1-44b2-9768-7d338c8f9485\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") "
Jul 01 18:52:54.508986 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.508924    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls\") pod \"c47711f2-01a1-44b2-9768-7d338c8f9485\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") "
Jul 01 18:52:54.508986 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.508945    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert\") pod \"c47711f2-01a1-44b2-9768-7d338c8f9485\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") "
Jul 01 18:52:54.508986 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.508974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlq8v\" (UniqueName: \"kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v\") pod \"c47711f2-01a1-44b2-9768-7d338c8f9485\" (UID: \"c47711f2-01a1-44b2-9768-7d338c8f9485\") "
Jul 01 18:52:54.509302 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.509258    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c47711f2-01a1-44b2-9768-7d338c8f9485" (UID: "c47711f2-01a1-44b2-9768-7d338c8f9485"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:52:54.509383 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.509299    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c47711f2-01a1-44b2-9768-7d338c8f9485" (UID: "c47711f2-01a1-44b2-9768-7d338c8f9485"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:54.509383 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.509360    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c47711f2-01a1-44b2-9768-7d338c8f9485" (UID: "c47711f2-01a1-44b2-9768-7d338c8f9485"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:54.510909 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.510890    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v" (OuterVolumeSpecName: "kube-api-access-vlq8v") pod "c47711f2-01a1-44b2-9768-7d338c8f9485" (UID: "c47711f2-01a1-44b2-9768-7d338c8f9485"). InnerVolumeSpecName "kube-api-access-vlq8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:52:54.510966 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.510937    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c47711f2-01a1-44b2-9768-7d338c8f9485" (UID: "c47711f2-01a1-44b2-9768-7d338c8f9485"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:52:54.610552 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.610503    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlq8v\" (UniqueName: \"kubernetes.io/projected/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-api-access-vlq8v\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:54.610552 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.610523    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c47711f2-01a1-44b2-9768-7d338c8f9485-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:54.610552 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.610532    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:54.610552 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.610542    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c47711f2-01a1-44b2-9768-7d338c8f9485-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:54.610552 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:54.610550    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c47711f2-01a1-44b2-9768-7d338c8f9485-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:52:55.422919 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.422889    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"
Jul 01 18:52:55.422919 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.422904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm" event={"ID":"c47711f2-01a1-44b2-9768-7d338c8f9485","Type":"ContainerDied","Data":"bef568db7861c8a982831c07e5b107ba9be3f864b245b0726a7ce6ad414b9d9b"}
Jul 01 18:52:55.423517 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.422968    2583 scope.go:117] "RemoveContainer" containerID="80dd8a94a7f99514a4f1f698fa7a119b8b3fbf3555e07ecafdd2e0ba87a9c07e"
Jul 01 18:52:55.431428 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.431405    2583 scope.go:117] "RemoveContainer" containerID="4eca312ed3400c2a4842d807bdbcf89d85549d95eee3c876943631fae7212274"
Jul 01 18:52:55.438367 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.438351    2583 scope.go:117] "RemoveContainer" containerID="6ae133a354223b397516e50a64bf1b85692a85b86a5d12af44108ac6e717ed77"
Jul 01 18:52:55.446754 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.446734    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:52:55.452514 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:55.452494    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e8497-predictor-76bbd98fd-bcnvm"]
Jul 01 18:52:56.365835 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:56.365807    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" path="/var/lib/kubelet/pods/ad26e5f8-d3c4-4b08-b958-006b7270464b/volumes"
Jul 01 18:52:56.366308 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:52:56.366278    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" path="/var/lib/kubelet/pods/c47711f2-01a1-44b2-9768-7d338c8f9485/volumes"
Jul 01 18:53:10.418395 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418362    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418646    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418657    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418667    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418672    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418680    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418686    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418691    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418697    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418707    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418711    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418720    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418726    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418733    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418738    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418746    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418751    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418757    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418762    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="storage-initializer"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418768    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418773    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418781    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418786    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418794    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418799    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418844    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418852    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418858    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47711f2-01a1-44b2-9768-7d338c8f9485" containerName="kserve-container"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418864    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418869    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6182d1-8191-45e8-bee8-ce37ec945e61" containerName="kube-rbac-proxy"
Jul 01 18:53:10.420927 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418875    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c5b5f9-c502-491d-a1c9-cbd415b2b0fe" containerName="kserve-container"
Jul 01 18:53:10.421928 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418882    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kube-rbac-proxy"
Jul 01 18:53:10.421928 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.418886    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad26e5f8-d3c4-4b08-b958-006b7270464b" containerName="kserve-container"
Jul 01 18:53:10.421928 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.421886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.424179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424153    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a0e25-predictor-serving-cert\""
Jul 01 18:53:10.424334 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a0e25-kube-rbac-proxy-sar-config\""
Jul 01 18:53:10.424334 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-64f2j\""
Jul 01 18:53:10.424334 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424249    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:53:10.424624 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:53:10.424716 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.424626    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:53:10.425029 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.425013    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:53:10.434509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.434489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:53:10.521610 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.521586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.521722 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.521622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4cd\" (UniqueName: \"kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.521722 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.521641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.521848 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.521726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.521848 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.521787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623112 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4cd\" (UniqueName: \"kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623278 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623652 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623623    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623923 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623903    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.623966 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.623903    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.625730 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.625701    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.632781 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.632755    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4cd\" (UniqueName: \"kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd\") pod \"isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.731992 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.731937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:10.861434 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:10.861348    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:53:10.863995 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:53:10.863944    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c80127e_92c0_48ad_8cd5_d9f629ecbf1d.slice/crio-bba327633100f17c3b124caa0a26036143fabc2f68d23f8ddd5e41591aae14d2 WatchSource:0}: Error finding container bba327633100f17c3b124caa0a26036143fabc2f68d23f8ddd5e41591aae14d2: Status 404 returned error can't find the container with id bba327633100f17c3b124caa0a26036143fabc2f68d23f8ddd5e41591aae14d2
Jul 01 18:53:11.490509 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:11.490477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerStarted","Data":"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"}
Jul 01 18:53:11.490970 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:11.490546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerStarted","Data":"bba327633100f17c3b124caa0a26036143fabc2f68d23f8ddd5e41591aae14d2"}
Jul 01 18:53:15.509195 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:15.509158    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerID="f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46" exitCode=0
Jul 01 18:53:15.509597 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:15.509211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerDied","Data":"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"}
Jul 01 18:53:16.515880 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.515843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerStarted","Data":"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"}
Jul 01 18:53:16.516245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.515906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerStarted","Data":"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"}
Jul 01 18:53:16.516245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.515930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerStarted","Data":"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"}
Jul 01 18:53:16.516245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.516220    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:16.516411 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.516386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:16.517634 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.517606    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:16.542709 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:16.542657    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podStartSLOduration=2.760177876 podStartE2EDuration="6.54264415s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782466274s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:53:16.540218341 +0000 UTC m=+954.555797939" watchObservedRunningTime="2026-07-01 18:53:16.54264415 +0000 UTC m=+954.558223695"
Jul 01 18:53:17.519875 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:17.519845    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:17.520264 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:17.519997    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:17.520998 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:17.520971    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:18.523124 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:18.523088    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:18.523524 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:18.523370    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:23.527935 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:23.527905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:53:23.528635 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:23.528601    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:23.528793 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:23.528772    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:33.529221 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:33.529175    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:33.529709 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:33.529689    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:43.528786 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:43.528746    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:43.529255 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:43.529199    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:53.528598 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:53.528510    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:53:53.529058 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:53:53.528975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:03.529269 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:03.529224    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:54:03.529682 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:03.529647    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:13.528495 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:13.528456    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:54:13.528976 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:13.528924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:23.528563 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:23.528524    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:54:23.528968 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:23.528945    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:33.529445 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:33.529419    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:54:33.530013 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:33.529986    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:54:45.715547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.715514    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:54:45.719749 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.719728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.721936 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.721910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-9bc99-predictor-serving-cert\""
Jul 01 18:54:45.722108 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.721915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-9bc99-kube-rbac-proxy-sar-config\""
Jul 01 18:54:45.734627 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.733365    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:54:45.740522 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.740490    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:54:45.740850 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.740803    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" containerID="cri-o://f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6" gracePeriod=30
Jul 01 18:54:45.740903 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.740874    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" containerID="cri-o://11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891" gracePeriod=30
Jul 01 18:54:45.740991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.740863    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" containerID="cri-o://1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1" gracePeriod=30
Jul 01 18:54:45.802032 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.802005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.802150 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.802047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.802150 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.802076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.802150 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.802096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7xf\" (UniqueName: \"kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.802256 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.802186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.864061 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.864038    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerID="11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891" exitCode=2
Jul 01 18:54:45.864173 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.864105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerDied","Data":"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"}
Jul 01 18:54:45.903164 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7xf\" (UniqueName: \"kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903274 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903616 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903598    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903832 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903816    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.903885 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.903867    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.905520 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.905500    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:45.911608 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:45.911589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7xf\" (UniqueName: \"kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf\") pod \"isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:46.029885 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:46.029833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:46.159057 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:46.159034    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:54:46.161208 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:54:46.161186    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93024dba_5b2f_4b3c_b741_0dadc98680f4.slice/crio-f6b809bd486625b2b5baedbcf50dadefd200889fb07f646de1fb4fa64f7639b8 WatchSource:0}: Error finding container f6b809bd486625b2b5baedbcf50dadefd200889fb07f646de1fb4fa64f7639b8: Status 404 returned error can't find the container with id f6b809bd486625b2b5baedbcf50dadefd200889fb07f646de1fb4fa64f7639b8
Jul 01 18:54:46.869411 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:46.869380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerStarted","Data":"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"}
Jul 01 18:54:46.869740 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:46.869458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerStarted","Data":"f6b809bd486625b2b5baedbcf50dadefd200889fb07f646de1fb4fa64f7639b8"}
Jul 01 18:54:48.524170 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:48.524134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:54:49.883011 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:49.882985    2583 generic.go:356] "Generic (PLEG): container finished" podID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerID="39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b" exitCode=0
Jul 01 18:54:49.883452 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:49.883055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerDied","Data":"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"}
Jul 01 18:54:49.885643 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:49.885621    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerID="f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6" exitCode=0
Jul 01 18:54:49.885746 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:49.885645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerDied","Data":"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"}
Jul 01 18:54:50.891345 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.891313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerStarted","Data":"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"}
Jul 01 18:54:50.891703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.891373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerStarted","Data":"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"}
Jul 01 18:54:50.891703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.891629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:50.891703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.891679    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:50.892952 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.892927    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:54:50.916414 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:50.916371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podStartSLOduration=2.257005292 podStartE2EDuration="5.916357977s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659352685s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:54:50.914476936 +0000 UTC m=+1048.930056537" watchObservedRunningTime="2026-07-01 18:54:50.916357977 +0000 UTC m=+1048.931937521"
Jul 01 18:54:51.895708 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:51.895664    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:54:53.523463 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:53.523420    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:54:53.528834 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:53.528804    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:54:53.530408 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:53.530385    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:56.899877 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:56.899852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:54:56.900436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:56.900409    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:54:58.523909 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:58.523874    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:54:58.524310 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:54:58.524010    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:55:03.524122 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:03.524085    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:55:03.528491 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:03.528456    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:55:03.531022 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:03.530998    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:55:06.900652 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:06.900608    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:08.523836 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:08.523800    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:55:13.523493 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:13.523450    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 18:55:13.528808 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:13.528782    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 18:55:13.528907 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:13.528899    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:55:13.530353 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:13.530319    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:55:13.530489 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:13.530419    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:55:15.892905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:15.892883    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:55:16.000847 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.000773    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerID="1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1" exitCode=0
Jul 01 18:55:16.000847 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.000827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerDied","Data":"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"}
Jul 01 18:55:16.000847 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.000843    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"
Jul 01 18:55:16.001057 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.000931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd" event={"ID":"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d","Type":"ContainerDied","Data":"bba327633100f17c3b124caa0a26036143fabc2f68d23f8ddd5e41591aae14d2"}
Jul 01 18:55:16.001057 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.000967    2583 scope.go:117] "RemoveContainer" containerID="1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"
Jul 01 18:55:16.008856 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.008840    2583 scope.go:117] "RemoveContainer" containerID="11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"
Jul 01 18:55:16.015816 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.015797    2583 scope.go:117] "RemoveContainer" containerID="f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"
Jul 01 18:55:16.016367 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config\") pod \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") "
Jul 01 18:55:16.016441 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016426    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location\") pod \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") "
Jul 01 18:55:16.016512 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l4cd\" (UniqueName: \"kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd\") pod \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") "
Jul 01 18:55:16.016561 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016525    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert\") pod \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") "
Jul 01 18:55:16.016601 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016558    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls\") pod \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\" (UID: \"6c80127e-92c0-48ad-8cd5-d9f629ecbf1d\") "
Jul 01 18:55:16.016788 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016745    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" (UID: "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:55:16.016839 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.016779    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" (UID: "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:55:16.017024 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.017005    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" (UID: "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:55:16.018525 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.018504    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd" (OuterVolumeSpecName: "kube-api-access-7l4cd") pod "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" (UID: "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d"). InnerVolumeSpecName "kube-api-access-7l4cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:55:16.018689 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.018675    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" (UID: "6c80127e-92c0-48ad-8cd5-d9f629ecbf1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:55:16.032408 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.032391    2583 scope.go:117] "RemoveContainer" containerID="f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"
Jul 01 18:55:16.039261 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.039243    2583 scope.go:117] "RemoveContainer" containerID="1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"
Jul 01 18:55:16.039570 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:55:16.039547    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1\": container with ID starting with 1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1 not found: ID does not exist" containerID="1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"
Jul 01 18:55:16.039635 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.039578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1"} err="failed to get container status \"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1\": rpc error: code = NotFound desc = could not find container \"1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1\": container with ID starting with 1cd660b10f00613d2186e576675304ca2aa44121e6923730724f0fd0ab6130a1 not found: ID does not exist"
Jul 01 18:55:16.039635 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.039593    2583 scope.go:117] "RemoveContainer" containerID="11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"
Jul 01 18:55:16.039865 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:55:16.039844    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891\": container with ID starting with 11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891 not found: ID does not exist" containerID="11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"
Jul 01 18:55:16.039913 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.039875    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891"} err="failed to get container status \"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891\": rpc error: code = NotFound desc = could not find container \"11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891\": container with ID starting with 11ef836c9535da0d771290d01e843c3590dfec8ab7974251ee7c17e16b11e891 not found: ID does not exist"
Jul 01 18:55:16.039913 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.039897    2583 scope.go:117] "RemoveContainer" containerID="f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"
Jul 01 18:55:16.040136 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:55:16.040116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6\": container with ID starting with f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6 not found: ID does not exist" containerID="f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"
Jul 01 18:55:16.040185 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.040142    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6"} err="failed to get container status \"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6\": rpc error: code = NotFound desc = could not find container \"f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6\": container with ID starting with f77e463d2a50707bf7ff862a7413c39c9b1af618b04d4006e7ebda9ab5b96fd6 not found: ID does not exist"
Jul 01 18:55:16.040185 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.040160    2583 scope.go:117] "RemoveContainer" containerID="f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"
Jul 01 18:55:16.040396 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:55:16.040377    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46\": container with ID starting with f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46 not found: ID does not exist" containerID="f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"
Jul 01 18:55:16.040445 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.040401    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46"} err="failed to get container status \"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46\": rpc error: code = NotFound desc = could not find container \"f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46\": container with ID starting with f6e3716d0cea38d58cc5fa08e741728d7acc59ac9553f1940d3073ac95e2fe46 not found: ID does not exist"
Jul 01 18:55:16.117277 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.117255    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:55:16.117277 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.117273    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:55:16.117277 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.117299    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:55:16.117465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.117308    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4cd\" (UniqueName: \"kubernetes.io/projected/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-kube-api-access-7l4cd\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:55:16.117465 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.117317    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:55:16.327104 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.327076    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:55:16.333569 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.333548    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a0e25-predictor-55588b5566-lj5kd"]
Jul 01 18:55:16.365691 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.365660    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" path="/var/lib/kubelet/pods/6c80127e-92c0-48ad-8cd5-d9f629ecbf1d/volumes"
Jul 01 18:55:16.900506 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:16.900477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:26.901344 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:26.901300    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:36.900494 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:36.900453    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:46.901036 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:46.900988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:56.900727 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:56.900687    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:55:59.361992 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:55:59.361951    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:09.362273 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:09.362230    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:19.362495 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:19.362452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:29.362223 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:29.362183    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:39.362202 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:39.362160    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:49.362385 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:49.362298    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:56:59.362769 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:56:59.362735    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:57:09.362653 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:09.362617    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:57:19.363054 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:19.363027    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:57:22.410186 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:22.410157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:57:22.411316 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:22.411295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 18:57:25.907433 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:25.907399    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:57:25.907814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:25.907599    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" containerID="cri-o://ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8" gracePeriod=30
Jul 01 18:57:25.907814 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:25.907648    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy" containerID="cri-o://0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09" gracePeriod=30
Jul 01 18:57:26.517198 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:26.517170    2583 generic.go:356] "Generic (PLEG): container finished" podID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerID="0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09" exitCode=2
Jul 01 18:57:26.517346 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:26.517240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerDied","Data":"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"}
Jul 01 18:57:26.896855 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:26.896825    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 01 18:57:29.362748 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:29.362695    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 18:57:31.030276 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030238    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:57:31.030759 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030739    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030766    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030780    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030788    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030801    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="storage-initializer"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030811    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="storage-initializer"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030826    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy"
Jul 01 18:57:31.030837 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030834    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy"
Jul 01 18:57:31.031179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030935    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="agent"
Jul 01 18:57:31.031179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030949    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kserve-container"
Jul 01 18:57:31.031179 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.030961    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c80127e-92c0-48ad-8cd5-d9f629ecbf1d" containerName="kube-rbac-proxy"
Jul 01 18:57:31.034394 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.034372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.036942 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.036921    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ff8cbb-predictor-serving-cert\""
Jul 01 18:57:31.037337 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.037322    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ff8cbb-kube-rbac-proxy-sar-config\""
Jul 01 18:57:31.045554 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.045531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:57:31.200146 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.200121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.200267 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.200156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.200267 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.200178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.200267 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.200215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwnz\" (UniqueName: \"kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.200267 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.200233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301040 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.300979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301040 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301040 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwnz\" (UniqueName: \"kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301230 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301230 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301230 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:57:31.301145    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-serving-cert: secret "isvc-primary-ff8cbb-predictor-serving-cert" not found
Jul 01 18:57:31.301230 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:57:31.301226    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls podName:47b8c81b-4896-4f7f-9f08-6ab55be9576f nodeName:}" failed. No retries permitted until 2026-07-01 18:57:31.801204357 +0000 UTC m=+1209.816783899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls") pod "isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f") : secret "isvc-primary-ff8cbb-predictor-serving-cert" not found
Jul 01 18:57:31.301466 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301451    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301848 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.301958 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.301819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.310034 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.310012    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwnz\" (UniqueName: \"kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.805303 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.805259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.807461 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.807435    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") pod \"isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") " pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:31.896085 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.896054    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 01 18:57:31.945069 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:31.945044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:32.072400 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:32.072328    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:57:32.076027 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:57:32.076000    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b8c81b_4896_4f7f_9f08_6ab55be9576f.slice/crio-74c85330e2ecde480f55f8f132a78b0ddde623dc470c7988ea3434cbc3f2d5a5 WatchSource:0}: Error finding container 74c85330e2ecde480f55f8f132a78b0ddde623dc470c7988ea3434cbc3f2d5a5: Status 404 returned error can't find the container with id 74c85330e2ecde480f55f8f132a78b0ddde623dc470c7988ea3434cbc3f2d5a5
Jul 01 18:57:32.550480 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:32.550434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerStarted","Data":"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"}
Jul 01 18:57:32.550645 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:32.550496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerStarted","Data":"74c85330e2ecde480f55f8f132a78b0ddde623dc470c7988ea3434cbc3f2d5a5"}
Jul 01 18:57:34.449217 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.449195    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:57:34.524247 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location\") pod \"93024dba-5b2f-4b3c-b741-0dadc98680f4\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") "
Jul 01 18:57:34.524247 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524229    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls\") pod \"93024dba-5b2f-4b3c-b741-0dadc98680f4\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") "
Jul 01 18:57:34.524436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config\") pod \"93024dba-5b2f-4b3c-b741-0dadc98680f4\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") "
Jul 01 18:57:34.524436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524299    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb7xf\" (UniqueName: \"kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf\") pod \"93024dba-5b2f-4b3c-b741-0dadc98680f4\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") "
Jul 01 18:57:34.524436 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert\") pod \"93024dba-5b2f-4b3c-b741-0dadc98680f4\" (UID: \"93024dba-5b2f-4b3c-b741-0dadc98680f4\") "
Jul 01 18:57:34.524600 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524535    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93024dba-5b2f-4b3c-b741-0dadc98680f4" (UID: "93024dba-5b2f-4b3c-b741-0dadc98680f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:57:34.524657 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524635    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93024dba-5b2f-4b3c-b741-0dadc98680f4" (UID: "93024dba-5b2f-4b3c-b741-0dadc98680f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:57:34.524770 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.524744    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93024dba-5b2f-4b3c-b741-0dadc98680f4" (UID: "93024dba-5b2f-4b3c-b741-0dadc98680f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:57:34.526177 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.526158    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93024dba-5b2f-4b3c-b741-0dadc98680f4" (UID: "93024dba-5b2f-4b3c-b741-0dadc98680f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:57:34.526279 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.526258    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf" (OuterVolumeSpecName: "kube-api-access-sb7xf") pod "93024dba-5b2f-4b3c-b741-0dadc98680f4" (UID: "93024dba-5b2f-4b3c-b741-0dadc98680f4"). InnerVolumeSpecName "kube-api-access-sb7xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:57:34.560321 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.560271    2583 generic.go:356] "Generic (PLEG): container finished" podID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerID="ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8" exitCode=0
Jul 01 18:57:34.560415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.560335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"
Jul 01 18:57:34.560415 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.560363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerDied","Data":"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"}
Jul 01 18:57:34.560490 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.560424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv" event={"ID":"93024dba-5b2f-4b3c-b741-0dadc98680f4","Type":"ContainerDied","Data":"f6b809bd486625b2b5baedbcf50dadefd200889fb07f646de1fb4fa64f7639b8"}
Jul 01 18:57:34.560490 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.560453    2583 scope.go:117] "RemoveContainer" containerID="0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"
Jul 01 18:57:34.569935 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.569921    2583 scope.go:117] "RemoveContainer" containerID="ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"
Jul 01 18:57:34.576876 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.576859    2583 scope.go:117] "RemoveContainer" containerID="39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"
Jul 01 18:57:34.583898 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.583878    2583 scope.go:117] "RemoveContainer" containerID="0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"
Jul 01 18:57:34.584119 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:57:34.584102    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09\": container with ID starting with 0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09 not found: ID does not exist" containerID="0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"
Jul 01 18:57:34.584170 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.584126    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09"} err="failed to get container status \"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09\": rpc error: code = NotFound desc = could not find container \"0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09\": container with ID starting with 0a3b8ba01987b6bd65e89679a19ce49fa89febb8327cdc856d2edf319087db09 not found: ID does not exist"
Jul 01 18:57:34.584170 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.584141    2583 scope.go:117] "RemoveContainer" containerID="ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"
Jul 01 18:57:34.584403 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:57:34.584387    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8\": container with ID starting with ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8 not found: ID does not exist" containerID="ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"
Jul 01 18:57:34.584463 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.584409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8"} err="failed to get container status \"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8\": rpc error: code = NotFound desc = could not find container \"ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8\": container with ID starting with ce7ce8a0021cb3c1256126aacc36f9473d437341f3725d6bd1101b33c2260ee8 not found: ID does not exist"
Jul 01 18:57:34.584463 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.584424    2583 scope.go:117] "RemoveContainer" containerID="39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"
Jul 01 18:57:34.584683 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:57:34.584663    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b\": container with ID starting with 39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b not found: ID does not exist" containerID="39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"
Jul 01 18:57:34.584729 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.584690    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b"} err="failed to get container status \"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b\": rpc error: code = NotFound desc = could not find container \"39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b\": container with ID starting with 39a592772c6d97be5b8ad4dda8f99343b10c9e26447ca076eea7ed07bbea233b not found: ID does not exist"
Jul 01 18:57:34.585575 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.585556    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:57:34.590491 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.590472    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-9bc99-predictor-588d48b7f6-bl2gv"]
Jul 01 18:57:34.625098 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.625079    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93024dba-5b2f-4b3c-b741-0dadc98680f4-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:57:34.625172 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.625100    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93024dba-5b2f-4b3c-b741-0dadc98680f4-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:57:34.625172 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.625110    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:57:34.625172 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.625118    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7xf\" (UniqueName: \"kubernetes.io/projected/93024dba-5b2f-4b3c-b741-0dadc98680f4-kube-api-access-sb7xf\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:57:34.625172 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:34.625127    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93024dba-5b2f-4b3c-b741-0dadc98680f4-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:57:36.366448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:36.366416    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" path="/var/lib/kubelet/pods/93024dba-5b2f-4b3c-b741-0dadc98680f4/volumes"
Jul 01 18:57:36.570785 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:36.570757    2583 generic.go:356] "Generic (PLEG): container finished" podID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerID="2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6" exitCode=0
Jul 01 18:57:36.570923 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:36.570801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerDied","Data":"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"}
Jul 01 18:57:37.577004 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.576976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerStarted","Data":"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"}
Jul 01 18:57:37.577360 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.577030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerStarted","Data":"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"}
Jul 01 18:57:37.577360 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.577311    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:37.577446 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.577429    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:37.578474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.578449    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:57:37.600500 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:37.600461    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podStartSLOduration=3.152882256 podStartE2EDuration="6.600450644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.447568388s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:57:37.599665674 +0000 UTC m=+1215.615245238" watchObservedRunningTime="2026-07-01 18:57:37.600450644 +0000 UTC m=+1215.616030188"
Jul 01 18:57:38.580922 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:38.580880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:57:43.585199 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:43.585169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:57:43.585786 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:43.585760    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:57:53.586639 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:57:53.586602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:03.586512 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:03.586475    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:13.586101 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:13.586062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:23.586016 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:23.585980    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:33.586706 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:33.586669    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:43.585841 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:43.585804    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 18:58:53.586459 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:58:53.586426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:59:01.226823 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.226789    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:01.227171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227137    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy"
Jul 01 18:59:01.227171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227150    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy"
Jul 01 18:59:01.227171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227162    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="storage-initializer"
Jul 01 18:59:01.227171 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227170    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="storage-initializer"
Jul 01 18:59:01.227326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227181    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container"
Jul 01 18:59:01.227326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227187    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container"
Jul 01 18:59:01.227326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227238    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kube-rbac-proxy"
Jul 01 18:59:01.227326 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.227249    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93024dba-5b2f-4b3c-b741-0dadc98680f4" containerName="kserve-container"
Jul 01 18:59:01.230739 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.230724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.233192 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.233171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ff8cbb-predictor-serving-cert\""
Jul 01 18:59:01.233339 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.233273    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ff8cbb\""
Jul 01 18:59:01.233339 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.233306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ff8cbb-dockercfg-zhj6c\""
Jul 01 18:59:01.233557 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.233542    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ff8cbb-kube-rbac-proxy-sar-config\""
Jul 01 18:59:01.243946 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.241068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:01.342251 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.342226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.342380 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.342262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.342380 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.342315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82ph\" (UniqueName: \"kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.342380 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.342347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.342380 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.342365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443347 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82ph\" (UniqueName: \"kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443448 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443597 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.443830 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.443806    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.444035 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.444018    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.444080 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.444033    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.445828 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.445808    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.451534 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.451511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82ph\" (UniqueName: \"kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph\") pod \"isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") " pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.544565 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.544508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:01.671825 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.671718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:01.674453 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:59:01.674424    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbe4017_f13a_42b0_99a8_2b11a8d3ca84.slice/crio-2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd WatchSource:0}: Error finding container 2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd: Status 404 returned error can't find the container with id 2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd
Jul 01 18:59:01.906868 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.906837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerStarted","Data":"bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"}
Jul 01 18:59:01.907041 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:01.906901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerStarted","Data":"2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd"}
Jul 01 18:59:04.921064 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:04.921044    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/0.log"
Jul 01 18:59:04.921430 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:04.921080    2583 generic.go:356] "Generic (PLEG): container finished" podID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerID="bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3" exitCode=1
Jul 01 18:59:04.921430 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:04.921162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerDied","Data":"bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"}
Jul 01 18:59:05.926811 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:05.926785    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/0.log"
Jul 01 18:59:05.927188 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:05.926872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerStarted","Data":"f1e6219949f942f44d1e55211bb754b0cf2970a43d4c418f82891b3222beb958"}
Jul 01 18:59:08.940038 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940009    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/1.log"
Jul 01 18:59:08.940469 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940440    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/0.log"
Jul 01 18:59:08.940514 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940470    2583 generic.go:356] "Generic (PLEG): container finished" podID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerID="f1e6219949f942f44d1e55211bb754b0cf2970a43d4c418f82891b3222beb958" exitCode=1
Jul 01 18:59:08.940567 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerDied","Data":"f1e6219949f942f44d1e55211bb754b0cf2970a43d4c418f82891b3222beb958"}
Jul 01 18:59:08.940622 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940611    2583 scope.go:117] "RemoveContainer" containerID="bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"
Jul 01 18:59:08.940934 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.940913    2583 scope.go:117] "RemoveContainer" containerID="bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"
Jul 01 18:59:08.951109 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:08.951088    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_kserve-ci-e2e-test(afbe4017-f13a-42b0-99a8-2b11a8d3ca84)\"" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84"
Jul 01 18:59:08.951310 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:08.951267    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_kserve-ci-e2e-test_afbe4017-f13a-42b0-99a8-2b11a8d3ca84_0 in pod sandbox 2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd from index: no such id: 'bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3'" containerID="bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"
Jul 01 18:59:08.951367 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:08.951321    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_kserve-ci-e2e-test_afbe4017-f13a-42b0-99a8-2b11a8d3ca84_0 in pod sandbox 2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd from index: no such id: 'bd62e7ad38e8a73fea62b11977ffdac176cd531c59f5d322c9bfbd7e041d76b3'"
Jul 01 18:59:09.946563 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:09.946534    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/1.log"
Jul 01 18:59:15.284189 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.284160    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:15.343089 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.341728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:59:15.343089 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.342035    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container" containerID="cri-o://b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7" gracePeriod=30
Jul 01 18:59:15.343089 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.342232    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kube-rbac-proxy" containerID="cri-o://ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e" gracePeriod=30
Jul 01 18:59:15.466017 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.465995    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/1.log"
Jul 01 18:59:15.468151 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.468133    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:15.651730 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.651698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls\") pod \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") "
Jul 01 18:59:15.651883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.651746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location\") pod \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") "
Jul 01 18:59:15.651883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.651773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f82ph\" (UniqueName: \"kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph\") pod \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") "
Jul 01 18:59:15.651883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.651800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert\") pod \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") "
Jul 01 18:59:15.651883 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.651820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config\") pod \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\" (UID: \"afbe4017-f13a-42b0-99a8-2b11a8d3ca84\") "
Jul 01 18:59:15.652115 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.652043    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afbe4017-f13a-42b0-99a8-2b11a8d3ca84" (UID: "afbe4017-f13a-42b0-99a8-2b11a8d3ca84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:59:15.652172 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.652151    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afbe4017-f13a-42b0-99a8-2b11a8d3ca84" (UID: "afbe4017-f13a-42b0-99a8-2b11a8d3ca84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:15.652228 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.652213    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afbe4017-f13a-42b0-99a8-2b11a8d3ca84" (UID: "afbe4017-f13a-42b0-99a8-2b11a8d3ca84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:15.653852 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.653832    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afbe4017-f13a-42b0-99a8-2b11a8d3ca84" (UID: "afbe4017-f13a-42b0-99a8-2b11a8d3ca84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:15.653852 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.653843    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph" (OuterVolumeSpecName: "kube-api-access-f82ph") pod "afbe4017-f13a-42b0-99a8-2b11a8d3ca84" (UID: "afbe4017-f13a-42b0-99a8-2b11a8d3ca84"). InnerVolumeSpecName "kube-api-access-f82ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:15.752425 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.752401    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:15.752425 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.752422    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:15.752562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.752431    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:15.752562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.752440    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82ph\" (UniqueName: \"kubernetes.io/projected/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-kube-api-access-f82ph\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:15.752562 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.752448    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afbe4017-f13a-42b0-99a8-2b11a8d3ca84-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:15.976805 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.976754    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx_afbe4017-f13a-42b0-99a8-2b11a8d3ca84/storage-initializer/1.log"
Jul 01 18:59:15.976902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.976836    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"
Jul 01 18:59:15.976902 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.976877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx" event={"ID":"afbe4017-f13a-42b0-99a8-2b11a8d3ca84","Type":"ContainerDied","Data":"2065b3ff6382be96b1164a4ce39f90cabd0b03dd3997b5695c4b660a64b682bd"}
Jul 01 18:59:15.977025 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.977009    2583 scope.go:117] "RemoveContainer" containerID="f1e6219949f942f44d1e55211bb754b0cf2970a43d4c418f82891b3222beb958"
Jul 01 18:59:15.979321 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.979273    2583 generic.go:356] "Generic (PLEG): container finished" podID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerID="ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e" exitCode=2
Jul 01 18:59:15.979422 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:15.979343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerDied","Data":"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"}
Jul 01 18:59:16.015070 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:16.015046    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:16.019619 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:16.019594    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff8cbb-predictor-6cb6ffbd68-7tkfx"]
Jul 01 18:59:16.366850 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:16.366821    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" path="/var/lib/kubelet/pods/afbe4017-f13a-42b0-99a8-2b11a8d3ca84/volumes"
Jul 01 18:59:18.581649 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.581610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 01 18:59:18.990051 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.990032    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:59:18.993674 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.993653    2583 generic.go:356] "Generic (PLEG): container finished" podID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerID="b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7" exitCode=0
Jul 01 18:59:18.993792 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.993694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerDied","Data":"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"}
Jul 01 18:59:18.993792 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.993699    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"
Jul 01 18:59:18.993792 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.993741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm" event={"ID":"47b8c81b-4896-4f7f-9f08-6ab55be9576f","Type":"ContainerDied","Data":"74c85330e2ecde480f55f8f132a78b0ddde623dc470c7988ea3434cbc3f2d5a5"}
Jul 01 18:59:18.993792 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:18.993769    2583 scope.go:117] "RemoveContainer" containerID="ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"
Jul 01 18:59:19.001194 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.001172    2583 scope.go:117] "RemoveContainer" containerID="b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"
Jul 01 18:59:19.007794 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.007780    2583 scope.go:117] "RemoveContainer" containerID="2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"
Jul 01 18:59:19.015710 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.015689    2583 scope.go:117] "RemoveContainer" containerID="ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"
Jul 01 18:59:19.016047 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:19.015995    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e\": container with ID starting with ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e not found: ID does not exist" containerID="ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"
Jul 01 18:59:19.016047 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.016033    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e"} err="failed to get container status \"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e\": rpc error: code = NotFound desc = could not find container \"ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e\": container with ID starting with ffdf3d58dfd91fc3fa87336afaa9436504e8e248180a54c87228f51e70b1366e not found: ID does not exist"
Jul 01 18:59:19.016187 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.016059    2583 scope.go:117] "RemoveContainer" containerID="b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"
Jul 01 18:59:19.016371 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:19.016346    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7\": container with ID starting with b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7 not found: ID does not exist" containerID="b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"
Jul 01 18:59:19.016474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.016380    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7"} err="failed to get container status \"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7\": rpc error: code = NotFound desc = could not find container \"b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7\": container with ID starting with b61a6e052497f0e83953d70e82b677752162c5254a0b60e04f5db1b87ed9acd7 not found: ID does not exist"
Jul 01 18:59:19.016474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.016404    2583 scope.go:117] "RemoveContainer" containerID="2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"
Jul 01 18:59:19.016672 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:19.016652    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6\": container with ID starting with 2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6 not found: ID does not exist" containerID="2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"
Jul 01 18:59:19.016726 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.016680    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6"} err="failed to get container status \"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6\": rpc error: code = NotFound desc = could not find container \"2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6\": container with ID starting with 2c3ad17adbe48bd7361eefef53f1aca5c8cf5e9533d2562812b29b9debfcfcc6 not found: ID does not exist"
Jul 01 18:59:19.178123 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config\") pod \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") "
Jul 01 18:59:19.178232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbwnz\" (UniqueName: \"kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz\") pod \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") "
Jul 01 18:59:19.178232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert\") pod \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") "
Jul 01 18:59:19.178232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178208    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location\") pod \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") "
Jul 01 18:59:19.178232 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") pod \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\" (UID: \"47b8c81b-4896-4f7f-9f08-6ab55be9576f\") "
Jul 01 18:59:19.178550 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178525    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47b8c81b-4896-4f7f-9f08-6ab55be9576f" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:19.178633 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178571    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47b8c81b-4896-4f7f-9f08-6ab55be9576f" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:59:19.178633 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.178583    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47b8c81b-4896-4f7f-9f08-6ab55be9576f" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:19.180210 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.180185    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz" (OuterVolumeSpecName: "kube-api-access-dbwnz") pod "47b8c81b-4896-4f7f-9f08-6ab55be9576f" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f"). InnerVolumeSpecName "kube-api-access-dbwnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:19.180313 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.180268    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47b8c81b-4896-4f7f-9f08-6ab55be9576f" (UID: "47b8c81b-4896-4f7f-9f08-6ab55be9576f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:19.278980 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.278959    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:19.278980 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.278980    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwnz\" (UniqueName: \"kubernetes.io/projected/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kube-api-access-dbwnz\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:19.279109 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.278989    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b8c81b-4896-4f7f-9f08-6ab55be9576f-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:19.279109 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.278997    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b8c81b-4896-4f7f-9f08-6ab55be9576f-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:19.279109 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.279007    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b8c81b-4896-4f7f-9f08-6ab55be9576f-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:19.317196 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.317171    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:59:19.321183 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:19.321164    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff8cbb-predictor-7bd8dcf59-bwkgm"]
Jul 01 18:59:20.366188 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.366146    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" path="/var/lib/kubelet/pods/47b8c81b-4896-4f7f-9f08-6ab55be9576f/volumes"
Jul 01 18:59:20.450027 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.449998    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:20.450329 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450331    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450340    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kube-rbac-proxy"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kube-rbac-proxy"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450358    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450364    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450377    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450382    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450389    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container"
Jul 01 18:59:20.450392 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450394    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container"
Jul 01 18:59:20.450679 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450439    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kube-rbac-proxy"
Jul 01 18:59:20.450679 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450451    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.450679 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450457    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b8c81b-4896-4f7f-9f08-6ab55be9576f" containerName="kserve-container"
Jul 01 18:59:20.450679 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.450463    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="afbe4017-f13a-42b0-99a8-2b11a8d3ca84" containerName="storage-initializer"
Jul 01 18:59:20.454892 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.454877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.457721 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:59:20.457836 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-659ab6-dockercfg-bjrb4\""
Jul 01 18:59:20.457905 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-659ab6-predictor-serving-cert\""
Jul 01 18:59:20.457957 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-659ab6\""
Jul 01 18:59:20.457957 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457917    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:59:20.457957 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.457924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-659ab6-kube-rbac-proxy-sar-config\""
Jul 01 18:59:20.458701 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.458686    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:59:20.465121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.465100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:20.489648 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.489597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.489648 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.489639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.489810 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.489714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.489810 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.489765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.489810 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.489800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh696\" (UniqueName: \"kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591021 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.590998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh696\" (UniqueName: \"kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591138 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591384 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:20.591200    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-serving-cert: secret "isvc-init-fail-659ab6-predictor-serving-cert" not found
Jul 01 18:59:20.591384 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:20.591260    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls podName:4f03cfbc-f3f5-4f34-b168-aa6afac91839 nodeName:}" failed. No retries permitted until 2026-07-01 18:59:21.091238823 +0000 UTC m=+1319.106818357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls") pod "isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839") : secret "isvc-init-fail-659ab6-predictor-serving-cert" not found
Jul 01 18:59:20.591542 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591523    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591733 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591715    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.591793 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.591740    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:20.599643 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:20.599623    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh696\" (UniqueName: \"kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:21.093922 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:21.093895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:21.096144 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:21.096125    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") pod \"isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") " pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:21.365416 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:21.365395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:21.499444 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:21.499406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:21.502193 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:59:21.502165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f03cfbc_f3f5_4f34_b168_aa6afac91839.slice/crio-d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223 WatchSource:0}: Error finding container d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223: Status 404 returned error can't find the container with id d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223
Jul 01 18:59:22.008130 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:22.008102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerStarted","Data":"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"}
Jul 01 18:59:22.008313 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:22.008155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerStarted","Data":"d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223"}
Jul 01 18:59:27.029009 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:27.028984    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/0.log"
Jul 01 18:59:27.029426 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:27.029019    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerID="0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4" exitCode=1
Jul 01 18:59:27.029426 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:27.029079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerDied","Data":"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"}
Jul 01 18:59:28.035991 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:28.035964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/0.log"
Jul 01 18:59:28.036403 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:28.036076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerStarted","Data":"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"}
Jul 01 18:59:30.494500 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:30.494468    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:30.494875 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:30.494651    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer" containerID="cri-o://cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e" gracePeriod=30
Jul 01 18:59:33.033396 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.033374    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/1.log"
Jul 01 18:59:33.033731 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.033717    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/0.log"
Jul 01 18:59:33.035972 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.035956    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:33.056117 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056095    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/1.log"
Jul 01 18:59:33.056460 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056444    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9_4f03cfbc-f3f5-4f34-b168-aa6afac91839/storage-initializer/0.log"
Jul 01 18:59:33.056547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056480    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerID="cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e" exitCode=1
Jul 01 18:59:33.056547 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056521    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"
Jul 01 18:59:33.056626 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerDied","Data":"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"}
Jul 01 18:59:33.056626 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9" event={"ID":"4f03cfbc-f3f5-4f34-b168-aa6afac91839","Type":"ContainerDied","Data":"d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223"}
Jul 01 18:59:33.056704 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.056648    2583 scope.go:117] "RemoveContainer" containerID="cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"
Jul 01 18:59:33.064878 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.064854    2583 scope.go:117] "RemoveContainer" containerID="0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"
Jul 01 18:59:33.071748 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.071734    2583 scope.go:117] "RemoveContainer" containerID="cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"
Jul 01 18:59:33.071969 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:33.071954    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e\": container with ID starting with cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e not found: ID does not exist" containerID="cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"
Jul 01 18:59:33.072009 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.071978    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e"} err="failed to get container status \"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e\": rpc error: code = NotFound desc = could not find container \"cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e\": container with ID starting with cd1486147a9179dc5faf946f94e0415ad4ca6f89a168182e710e0355a1dc6c3e not found: ID does not exist"
Jul 01 18:59:33.072009 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.071993    2583 scope.go:117] "RemoveContainer" containerID="0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"
Jul 01 18:59:33.072193 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:33.072177    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4\": container with ID starting with 0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4 not found: ID does not exist" containerID="0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"
Jul 01 18:59:33.072233 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.072202    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4"} err="failed to get container status \"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4\": rpc error: code = NotFound desc = could not find container \"0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4\": container with ID starting with 0bdc2d9d14651fe1495459d4e1055ac5c2a1b605c59d236fa23450301ef349c4 not found: ID does not exist"
Jul 01 18:59:33.083502 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert\") pod \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") "
Jul 01 18:59:33.083585 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config\") pod \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") "
Jul 01 18:59:33.083585 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") pod \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") "
Jul 01 18:59:33.083685 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location\") pod \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") "
Jul 01 18:59:33.083743 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh696\" (UniqueName: \"kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696\") pod \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\" (UID: \"4f03cfbc-f3f5-4f34-b168-aa6afac91839\") "
Jul 01 18:59:33.083854 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083832    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f03cfbc-f3f5-4f34-b168-aa6afac91839" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:33.083924 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083870    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f03cfbc-f3f5-4f34-b168-aa6afac91839" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:59:33.083924 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083882    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f03cfbc-f3f5-4f34-b168-aa6afac91839" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:33.084019 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.083935    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:33.085462 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.085436    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f03cfbc-f3f5-4f34-b168-aa6afac91839" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:33.085566 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.085545    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696" (OuterVolumeSpecName: "kube-api-access-lh696") pod "4f03cfbc-f3f5-4f34-b168-aa6afac91839" (UID: "4f03cfbc-f3f5-4f34-b168-aa6afac91839"). InnerVolumeSpecName "kube-api-access-lh696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:33.184245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.184195    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:33.184245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.184215    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f03cfbc-f3f5-4f34-b168-aa6afac91839-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:33.184245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.184223    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:33.184245 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.184234    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh696\" (UniqueName: \"kubernetes.io/projected/4f03cfbc-f3f5-4f34-b168-aa6afac91839-kube-api-access-lh696\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 18:59:33.384092 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:33.384063    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f03cfbc_f3f5_4f34_b168_aa6afac91839.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f03cfbc_f3f5_4f34_b168_aa6afac91839.slice/crio-d20ef01fe1422b437251bfa06a6da0c32f8a622f8108f596e55ebe7213c5c223\": RecentStats: unable to find data in memory cache]"
Jul 01 18:59:33.399227 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.399207    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:33.403514 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:33.403495    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-659ab6-predictor-7bd487c844-8xxv9"]
Jul 01 18:59:34.366241 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:34.366204    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" path="/var/lib/kubelet/pods/4f03cfbc-f3f5-4f34-b168-aa6afac91839/volumes"
Jul 01 18:59:35.675720 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.675693    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 18:59:35.676121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676003    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.676121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676014    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.676121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676027    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.676121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676032    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.676121 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676086    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.676381 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.676237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f03cfbc-f3f5-4f34-b168-aa6afac91839" containerName="storage-initializer"
Jul 01 18:59:35.680663 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.680647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.683474 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683447    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-267eb-predictor-serving-cert\""
Jul 01 18:59:35.683618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683458    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:59:35.683618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:59:35.683618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:59:35.683618 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683595    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-64f2j\""
Jul 01 18:59:35.683796 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.683624    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:59:35.684458 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.684440    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-267eb-kube-rbac-proxy-sar-config\""
Jul 01 18:59:35.692442 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.692425    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 18:59:35.700239 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.700222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.700359 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.700246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.700359 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.700279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.700359 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.700313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbbm\" (UniqueName: \"kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.700484 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.700387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.801703 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.801678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.801827 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.801708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.801827 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.801738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.801827 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.801753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbbm\" (UniqueName: \"kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.801827 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.801772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.802059 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:35.801891    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-267eb-predictor-serving-cert: secret "raw-sklearn-267eb-predictor-serving-cert" not found
Jul 01 18:59:35.802059 ip-10-0-137-241 kubenswrapper[2583]: E0701 18:59:35.801972    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls podName:be98b8fb-f880-45ba-aad2-8dbe747fd632 nodeName:}" failed. No retries permitted until 2026-07-01 18:59:36.301950537 +0000 UTC m=+1334.317530076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls") pod "raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632") : secret "raw-sklearn-267eb-predictor-serving-cert" not found
Jul 01 18:59:35.802156 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.802145    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.802417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.802398    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.802493 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.802465    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:35.812730 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:35.812711    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbbm\" (UniqueName: \"kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:36.305305 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:36.305259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:36.307550 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:36.307529    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") pod \"raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") " pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:36.591390 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:36.591324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:36.720120 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:36.720096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 18:59:36.722874 ip-10-0-137-241 kubenswrapper[2583]: W0701 18:59:36.722849    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe98b8fb_f880_45ba_aad2_8dbe747fd632.slice/crio-31e17cd7b940156fe5f7b10a58745d2cc2d224de1541f2229b703015ec20cd47 WatchSource:0}: Error finding container 31e17cd7b940156fe5f7b10a58745d2cc2d224de1541f2229b703015ec20cd47: Status 404 returned error can't find the container with id 31e17cd7b940156fe5f7b10a58745d2cc2d224de1541f2229b703015ec20cd47
Jul 01 18:59:37.076272 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:37.076239    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerStarted","Data":"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"}
Jul 01 18:59:37.076450 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:37.076316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerStarted","Data":"31e17cd7b940156fe5f7b10a58745d2cc2d224de1541f2229b703015ec20cd47"}
Jul 01 18:59:41.093417 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:41.093329    2583 generic.go:356] "Generic (PLEG): container finished" podID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerID="28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b" exitCode=0
Jul 01 18:59:41.093884 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:41.093409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerDied","Data":"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"}
Jul 01 18:59:42.099533 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:42.099497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerStarted","Data":"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"}
Jul 01 18:59:42.099916 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:42.099559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerStarted","Data":"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"}
Jul 01 18:59:42.099916 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:42.099855    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:42.122145 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:42.122101    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podStartSLOduration=3.229232879 podStartE2EDuration="7.122088647s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.892855768s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:59:42.120477583 +0000 UTC m=+1340.136057128" watchObservedRunningTime="2026-07-01 18:59:42.122088647 +0000 UTC m=+1340.137668192"
Jul 01 18:59:43.103764 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:43.103738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:43.104938 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:43.104909    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 18:59:44.107230 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:44.107197    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 18:59:49.111970 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:49.111899    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 18:59:49.112402 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:49.112376    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 18:59:59.112772 ip-10-0-137-241 kubenswrapper[2583]: I0701 18:59:59.112731    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:09.113131 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:09.113096    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:19.113059 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:19.113015    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:29.113241 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:29.113200    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:39.112343 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:39.112304    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:49.113098 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:49.113062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:00:59.113916 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:00:59.113880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 19:01:05.841842 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:05.841811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 19:01:05.842427 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:05.842150    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" containerID="cri-o://bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590" gracePeriod=30
Jul 01 19:01:05.842427 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:05.842160    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kube-rbac-proxy" containerID="cri-o://35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b" gracePeriod=30
Jul 01 19:01:06.439636 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:06.439601    2583 generic.go:356] "Generic (PLEG): container finished" podID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerID="35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b" exitCode=2
Jul 01 19:01:06.439803 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:06.439678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerDied","Data":"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"}
Jul 01 19:01:09.107674 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:09.107635    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 01 19:01:09.113128 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:09.113103    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 19:01:10.291104 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.291076    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 19:01:10.400137 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config\") pod \"be98b8fb-f880-45ba-aad2-8dbe747fd632\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") "
Jul 01 19:01:10.400339 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert\") pod \"be98b8fb-f880-45ba-aad2-8dbe747fd632\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") "
Jul 01 19:01:10.400339 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400246    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfbbm\" (UniqueName: \"kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm\") pod \"be98b8fb-f880-45ba-aad2-8dbe747fd632\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") "
Jul 01 19:01:10.400339 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400305    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location\") pod \"be98b8fb-f880-45ba-aad2-8dbe747fd632\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") "
Jul 01 19:01:10.400525 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") pod \"be98b8fb-f880-45ba-aad2-8dbe747fd632\" (UID: \"be98b8fb-f880-45ba-aad2-8dbe747fd632\") "
Jul 01 19:01:10.400586 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400545    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be98b8fb-f880-45ba-aad2-8dbe747fd632" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:01:10.400643 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400585    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be98b8fb-f880-45ba-aad2-8dbe747fd632" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:01:10.400643 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400612    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be98b8fb-f880-45ba-aad2-8dbe747fd632" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:01:10.400747 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400682    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98b8fb-f880-45ba-aad2-8dbe747fd632-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:01:10.400747 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400699    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:01:10.400747 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.400713    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98b8fb-f880-45ba-aad2-8dbe747fd632-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:01:10.402447 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.402426    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm" (OuterVolumeSpecName: "kube-api-access-zfbbm") pod "be98b8fb-f880-45ba-aad2-8dbe747fd632" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632"). InnerVolumeSpecName "kube-api-access-zfbbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:01:10.402515 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.402448    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be98b8fb-f880-45ba-aad2-8dbe747fd632" (UID: "be98b8fb-f880-45ba-aad2-8dbe747fd632"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:01:10.457011 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.456979    2583 generic.go:356] "Generic (PLEG): container finished" podID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerID="bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590" exitCode=0
Jul 01 19:01:10.457156 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.457029    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"
Jul 01 19:01:10.457156 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.457075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerDied","Data":"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"}
Jul 01 19:01:10.457156 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.457122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m" event={"ID":"be98b8fb-f880-45ba-aad2-8dbe747fd632","Type":"ContainerDied","Data":"31e17cd7b940156fe5f7b10a58745d2cc2d224de1541f2229b703015ec20cd47"}
Jul 01 19:01:10.457156 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.457147    2583 scope.go:117] "RemoveContainer" containerID="35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"
Jul 01 19:01:10.465511 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.465492    2583 scope.go:117] "RemoveContainer" containerID="bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"
Jul 01 19:01:10.472748 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.472733    2583 scope.go:117] "RemoveContainer" containerID="28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"
Jul 01 19:01:10.480116 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480101    2583 scope.go:117] "RemoveContainer" containerID="35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"
Jul 01 19:01:10.480381 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:01:10.480364    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b\": container with ID starting with 35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b not found: ID does not exist" containerID="35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"
Jul 01 19:01:10.480437 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480389    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b"} err="failed to get container status \"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b\": rpc error: code = NotFound desc = could not find container \"35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b\": container with ID starting with 35b5d181c1609c0f01cfca95c9a6389d7fecf679e9dc1fb08bd7fd48133cd70b not found: ID does not exist"
Jul 01 19:01:10.480437 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480409    2583 scope.go:117] "RemoveContainer" containerID="bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"
Jul 01 19:01:10.480627 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:01:10.480612    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590\": container with ID starting with bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590 not found: ID does not exist" containerID="bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"
Jul 01 19:01:10.480675 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480630    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590"} err="failed to get container status \"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590\": rpc error: code = NotFound desc = could not find container \"bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590\": container with ID starting with bf93d14e86d4e20bfd891b12145a14cab844fa36f2ce85e478f9b070d0882590 not found: ID does not exist"
Jul 01 19:01:10.480675 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480642    2583 scope.go:117] "RemoveContainer" containerID="28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"
Jul 01 19:01:10.480675 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480651    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 19:01:10.480864 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:01:10.480848    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b\": container with ID starting with 28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b not found: ID does not exist" containerID="28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"
Jul 01 19:01:10.480906 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.480869    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b"} err="failed to get container status \"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b\": rpc error: code = NotFound desc = could not find container \"28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b\": container with ID starting with 28b793611997573857d610c39f21307be2c35793a702d41917d5945eae04ac2b not found: ID does not exist"
Jul 01 19:01:10.485094 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.485078    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-267eb-predictor-6cc7b567c8-fsl8m"]
Jul 01 19:01:10.501806 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.501788    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfbbm\" (UniqueName: \"kubernetes.io/projected/be98b8fb-f880-45ba-aad2-8dbe747fd632-kube-api-access-zfbbm\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:01:10.501806 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.501807    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98b8fb-f880-45ba-aad2-8dbe747fd632-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:01:10.982584 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982553    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:01:10.982925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982904    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="storage-initializer"
Jul 01 19:01:10.982925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982919    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="storage-initializer"
Jul 01 19:01:10.982925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982926    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container"
Jul 01 19:01:10.982925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982932    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container"
Jul 01 19:01:10.983217 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kube-rbac-proxy"
Jul 01 19:01:10.983217 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kube-rbac-proxy"
Jul 01 19:01:10.983217 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.982994    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kserve-container"
Jul 01 19:01:10.983217 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.983003    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" containerName="kube-rbac-proxy"
Jul 01 19:01:10.987493 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.987475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:10.989943 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.989924    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:01:10.990043 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.989971    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-0e8fc-predictor-serving-cert\""
Jul 01 19:01:10.990111 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.990082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-64f2j\""
Jul 01 19:01:10.990200 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.990183    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-0e8fc-kube-rbac-proxy-sar-config\""
Jul 01 19:01:10.990266 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.990256    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:01:10.990347 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.990301    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:01:10.990399 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.990368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:01:10.997037 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:10.997014    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:01:11.106088 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.106051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.106088 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.106094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.106360 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.106158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9cd\" (UniqueName: \"kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.106360 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.106209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.106360 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.106247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.206823 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.206782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207002 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.206836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207067 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.206996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207067 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.207043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9cd\" (UniqueName: \"kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207178 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.207094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207365 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.207344    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207605 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.207585    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.207700 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.207589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.209453 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.209435    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.216082 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.216061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9cd\" (UniqueName: \"kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd\") pod \"raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.298494 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.298437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:11.431554 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.429782    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:01:11.435225 ip-10-0-137-241 kubenswrapper[2583]: W0701 19:01:11.435198    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92fc535_be9d_49c5_ae00_5faae95ef1dd.slice/crio-de0fa61847ae566b320477b2afe4437535697a2939c7d3a9e608ef6dbd639f8e WatchSource:0}: Error finding container de0fa61847ae566b320477b2afe4437535697a2939c7d3a9e608ef6dbd639f8e: Status 404 returned error can't find the container with id de0fa61847ae566b320477b2afe4437535697a2939c7d3a9e608ef6dbd639f8e
Jul 01 19:01:11.461875 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:11.461846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerStarted","Data":"de0fa61847ae566b320477b2afe4437535697a2939c7d3a9e608ef6dbd639f8e"}
Jul 01 19:01:12.368754 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:12.368722    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be98b8fb-f880-45ba-aad2-8dbe747fd632" path="/var/lib/kubelet/pods/be98b8fb-f880-45ba-aad2-8dbe747fd632/volumes"
Jul 01 19:01:12.467273 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:12.467228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerStarted","Data":"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"}
Jul 01 19:01:15.480865 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:15.480832    2583 generic.go:356] "Generic (PLEG): container finished" podID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerID="8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b" exitCode=0
Jul 01 19:01:15.481241 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:15.480905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerDied","Data":"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"}
Jul 01 19:01:16.486414 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:16.486382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerStarted","Data":"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"}
Jul 01 19:01:16.486769 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:16.486445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerStarted","Data":"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"}
Jul 01 19:01:16.486769 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:16.486632    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:16.509093 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:16.509044    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podStartSLOduration=2.90639402 podStartE2EDuration="6.509029118s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.602635098s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:01:16.507340465 +0000 UTC m=+1434.522920025" watchObservedRunningTime="2026-07-01 19:01:16.509029118 +0000 UTC m=+1434.524608662"
Jul 01 19:01:17.490707 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:17.490680    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:17.491833 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:17.491810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:01:18.494596 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:18.494546    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:01:23.500641 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:23.500615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:01:23.501077 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:23.501054    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:01:33.501159 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:33.501121    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:01:43.501651 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:43.501613    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:01:53.501511 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:01:53.501471    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:02:03.501322 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:03.501270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:02:13.501487 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:13.501453    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:02:22.437689 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:22.437665    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 19:02:22.439034 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:22.439011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 19:02:23.501046 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:23.501009    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:02:33.501997 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:33.501970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:02:41.093473 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:41.093439    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:02:41.093850 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:41.093686    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" containerID="cri-o://5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1" gracePeriod=30
Jul 01 19:02:41.093850 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:41.093810    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kube-rbac-proxy" containerID="cri-o://bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34" gracePeriod=30
Jul 01 19:02:41.820976 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:41.820941    2583 generic.go:356] "Generic (PLEG): container finished" podID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerID="bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34" exitCode=2
Jul 01 19:02:41.821144 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:41.821018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerDied","Data":"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"}
Jul 01 19:02:43.495741 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:43.495707    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 01 19:02:43.501114 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:43.501088    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 19:02:45.046523 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.046502    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:02:45.129740 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.129710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls\") pod \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") "
Jul 01 19:02:45.129892 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.129747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert\") pod \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") "
Jul 01 19:02:45.129892 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.129805    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config\") pod \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") "
Jul 01 19:02:45.129892 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.129849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location\") pod \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") "
Jul 01 19:02:45.129892 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.129889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td9cd\" (UniqueName: \"kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd\") pod \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\" (UID: \"d92fc535-be9d-49c5-ae00-5faae95ef1dd\") "
Jul 01 19:02:45.130224 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.130200    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d92fc535-be9d-49c5-ae00-5faae95ef1dd" (UID: "d92fc535-be9d-49c5-ae00-5faae95ef1dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:02:45.130280 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.130224    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d92fc535-be9d-49c5-ae00-5faae95ef1dd" (UID: "d92fc535-be9d-49c5-ae00-5faae95ef1dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:02:45.130280 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.130224    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d92fc535-be9d-49c5-ae00-5faae95ef1dd" (UID: "d92fc535-be9d-49c5-ae00-5faae95ef1dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:02:45.131902 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.131873    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd" (OuterVolumeSpecName: "kube-api-access-td9cd") pod "d92fc535-be9d-49c5-ae00-5faae95ef1dd" (UID: "d92fc535-be9d-49c5-ae00-5faae95ef1dd"). InnerVolumeSpecName "kube-api-access-td9cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:02:45.132003 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.131926    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d92fc535-be9d-49c5-ae00-5faae95ef1dd" (UID: "d92fc535-be9d-49c5-ae00-5faae95ef1dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:02:45.230659 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.230640    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92fc535-be9d-49c5-ae00-5faae95ef1dd-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:02:45.230659 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.230659    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:02:45.230796 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.230670    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:02:45.230796 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.230679    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:02:45.230796 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.230688    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9cd\" (UniqueName: \"kubernetes.io/projected/d92fc535-be9d-49c5-ae00-5faae95ef1dd-kube-api-access-td9cd\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:02:45.840638 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.840604    2583 generic.go:356] "Generic (PLEG): container finished" podID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerID="5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1" exitCode=0
Jul 01 19:02:45.840826 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.840657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerDied","Data":"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"}
Jul 01 19:02:45.840826 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.840664    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"
Jul 01 19:02:45.840826 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.840709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw" event={"ID":"d92fc535-be9d-49c5-ae00-5faae95ef1dd","Type":"ContainerDied","Data":"de0fa61847ae566b320477b2afe4437535697a2939c7d3a9e608ef6dbd639f8e"}
Jul 01 19:02:45.840826 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.840738    2583 scope.go:117] "RemoveContainer" containerID="bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"
Jul 01 19:02:45.849577 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.849554    2583 scope.go:117] "RemoveContainer" containerID="5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"
Jul 01 19:02:45.856925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.856908    2583 scope.go:117] "RemoveContainer" containerID="8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"
Jul 01 19:02:45.864119 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.864098    2583 scope.go:117] "RemoveContainer" containerID="bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"
Jul 01 19:02:45.864529 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:02:45.864509    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34\": container with ID starting with bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34 not found: ID does not exist" containerID="bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"
Jul 01 19:02:45.864627 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.864539    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34"} err="failed to get container status \"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34\": rpc error: code = NotFound desc = could not find container \"bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34\": container with ID starting with bc4ec09e32c261f615b05c4e2f3aa9e87ed3ede31846cf553523f2cf756a4c34 not found: ID does not exist"
Jul 01 19:02:45.864627 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.864562    2583 scope.go:117] "RemoveContainer" containerID="5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"
Jul 01 19:02:45.864799 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:02:45.864782    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1\": container with ID starting with 5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1 not found: ID does not exist" containerID="5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"
Jul 01 19:02:45.864873 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.864807    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1"} err="failed to get container status \"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1\": rpc error: code = NotFound desc = could not find container \"5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1\": container with ID starting with 5a5b0e961f69279375ac6918959f35d6aa4f9ac1bd4f9aee62f0987e867542b1 not found: ID does not exist"
Jul 01 19:02:45.864873 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.864829    2583 scope.go:117] "RemoveContainer" containerID="8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"
Jul 01 19:02:45.865086 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:02:45.865067    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b\": container with ID starting with 8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b not found: ID does not exist" containerID="8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"
Jul 01 19:02:45.865125 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.865095    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b"} err="failed to get container status \"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b\": rpc error: code = NotFound desc = could not find container \"8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b\": container with ID starting with 8db7dc562e0359869216a6ca527cf605e4caf8cbe9286c522b7580f52d5e840b not found: ID does not exist"
Jul 01 19:02:45.865189 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.865173    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:02:45.870300 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:45.870261    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-0e8fc-predictor-5d9f6785c4-2g9vw"]
Jul 01 19:02:46.370226 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:46.370190    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" path="/var/lib/kubelet/pods/d92fc535-be9d-49c5-ae00-5faae95ef1dd/volumes"
Jul 01 19:02:47.149759 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.149726    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m67ws/must-gather-hddln"]
Jul 01 19:02:47.150104 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150088    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kube-rbac-proxy"
Jul 01 19:02:47.150194 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kube-rbac-proxy"
Jul 01 19:02:47.150194 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150128    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container"
Jul 01 19:02:47.150194 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150136    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container"
Jul 01 19:02:47.150194 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="storage-initializer"
Jul 01 19:02:47.150194 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="storage-initializer"
Jul 01 19:02:47.150531 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150367    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kserve-container"
Jul 01 19:02:47.150531 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.150384    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92fc535-be9d-49c5-ae00-5faae95ef1dd" containerName="kube-rbac-proxy"
Jul 01 19:02:47.154797 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.154777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.157437 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.157419    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m67ws\"/\"kube-root-ca.crt\""
Jul 01 19:02:47.158221 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.158203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m67ws\"/\"openshift-service-ca.crt\""
Jul 01 19:02:47.158221 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.158214    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m67ws\"/\"default-dockercfg-dvwtl\""
Jul 01 19:02:47.160695 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.160674    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m67ws/must-gather-hddln"]
Jul 01 19:02:47.245670 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.245645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czstb\" (UniqueName: \"kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.245783 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.245676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.346723 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.346694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czstb\" (UniqueName: \"kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.346826 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.346732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.347096 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.347081    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.356071 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.356043    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czstb\" (UniqueName: \"kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb\") pod \"must-gather-hddln\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") " pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.476009 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.475937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:02:47.604468 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.604395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m67ws/must-gather-hddln"]
Jul 01 19:02:47.607103 ip-10-0-137-241 kubenswrapper[2583]: W0701 19:02:47.607077    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf578bf_a994_4e7e_923e_ed3ab5e7e08d.slice/crio-966d921281b31ede88e5800f8a5d9d1c3b3ed44f55c40cef16719efb538408dd WatchSource:0}: Error finding container 966d921281b31ede88e5800f8a5d9d1c3b3ed44f55c40cef16719efb538408dd: Status 404 returned error can't find the container with id 966d921281b31ede88e5800f8a5d9d1c3b3ed44f55c40cef16719efb538408dd
Jul 01 19:02:47.608766 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.608751    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:02:47.850042 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:47.849978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m67ws/must-gather-hddln" event={"ID":"abf578bf-a994-4e7e-923e-ed3ab5e7e08d","Type":"ContainerStarted","Data":"966d921281b31ede88e5800f8a5d9d1c3b3ed44f55c40cef16719efb538408dd"}
Jul 01 19:02:53.880400 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:53.880361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m67ws/must-gather-hddln" event={"ID":"abf578bf-a994-4e7e-923e-ed3ab5e7e08d","Type":"ContainerStarted","Data":"ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6"}
Jul 01 19:02:53.880787 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:53.880417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m67ws/must-gather-hddln" event={"ID":"abf578bf-a994-4e7e-923e-ed3ab5e7e08d","Type":"ContainerStarted","Data":"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"}
Jul 01 19:02:53.901462 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:02:53.901416    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m67ws/must-gather-hddln" podStartSLOduration=0.911513018 podStartE2EDuration="6.90140169s" totalImagesPullingTime="5.989888672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:02:53.900443222 +0000 UTC m=+1531.916022770" watchObservedRunningTime="2026-07-01 19:02:53.90140169 +0000 UTC m=+1531.916981234"
Jul 01 19:03:14.974692 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:14.974664    2583 generic.go:356] "Generic (PLEG): container finished" podID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerID="d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5" exitCode=0
Jul 01 19:03:14.975037 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:14.974742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m67ws/must-gather-hddln" event={"ID":"abf578bf-a994-4e7e-923e-ed3ab5e7e08d","Type":"ContainerDied","Data":"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"}
Jul 01 19:03:14.975037 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:14.975016    2583 scope.go:117] "RemoveContainer" containerID="d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"
Jul 01 19:03:15.950340 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:15.950305    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m67ws_must-gather-hddln_abf578bf-a994-4e7e-923e-ed3ab5e7e08d/gather/0.log"
Jul 01 19:03:19.185719 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:19.185686    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t5z88_048bfc0b-24ff-4316-8586-8a828d189524/global-pull-secret-syncer/0.log"
Jul 01 19:03:19.388398 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:19.388370    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-spjhz_53ae8699-9280-465a-a248-6eade14246da/konnectivity-agent/0.log"
Jul 01 19:03:19.414677 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:19.414648    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-241.ec2.internal_e08fe03195acc53e90bd3f1c6ffa3dc8/haproxy/0.log"
Jul 01 19:03:21.389979 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.389944    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-m67ws/must-gather-hddln"]
Jul 01 19:03:21.390443 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.390190    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-m67ws/must-gather-hddln" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="copy" containerID="cri-o://ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6" gracePeriod=2
Jul 01 19:03:21.392331 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:21.392278    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hddln\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m67ws\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:21.392894 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.392870    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-m67ws/must-gather-hddln"]
Jul 01 19:03:21.621407 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.621386    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m67ws_must-gather-hddln_abf578bf-a994-4e7e-923e-ed3ab5e7e08d/copy/0.log"
Jul 01 19:03:21.622978 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.622962    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:21.625111 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:21.625087    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hddln\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m67ws\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:21.725356 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.725304    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output\") pod \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") "
Jul 01 19:03:21.725459 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.725383    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czstb\" (UniqueName: \"kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb\") pod \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\" (UID: \"abf578bf-a994-4e7e-923e-ed3ab5e7e08d\") "
Jul 01 19:03:21.726671 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.726650    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "abf578bf-a994-4e7e-923e-ed3ab5e7e08d" (UID: "abf578bf-a994-4e7e-923e-ed3ab5e7e08d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:03:21.727325 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.727307    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb" (OuterVolumeSpecName: "kube-api-access-czstb") pod "abf578bf-a994-4e7e-923e-ed3ab5e7e08d" (UID: "abf578bf-a994-4e7e-923e-ed3ab5e7e08d"). InnerVolumeSpecName "kube-api-access-czstb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:03:21.826207 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.826186    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czstb\" (UniqueName: \"kubernetes.io/projected/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-kube-api-access-czstb\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:03:21.826207 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:21.826206    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abf578bf-a994-4e7e-923e-ed3ab5e7e08d-must-gather-output\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Jul 01 19:03:22.002707 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.002648    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m67ws_must-gather-hddln_abf578bf-a994-4e7e-923e-ed3ab5e7e08d/copy/0.log"
Jul 01 19:03:22.003058 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.003032    2583 generic.go:356] "Generic (PLEG): container finished" podID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerID="ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6" exitCode=143
Jul 01 19:03:22.003152 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.003083    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:22.003152 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.003132    2583 scope.go:117] "RemoveContainer" containerID="ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6"
Jul 01 19:03:22.005756 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:22.005729    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hddln\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m67ws\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:22.012297 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.012262    2583 scope.go:117] "RemoveContainer" containerID="d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"
Jul 01 19:03:22.014174 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:22.014153    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hddln\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m67ws\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:22.023491 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.023476    2583 scope.go:117] "RemoveContainer" containerID="ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6"
Jul 01 19:03:22.023738 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:22.023721    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6\": container with ID starting with ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6 not found: ID does not exist" containerID="ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6"
Jul 01 19:03:22.023777 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.023749    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6"} err="failed to get container status \"ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6\": rpc error: code = NotFound desc = could not find container \"ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6\": container with ID starting with ee5e52ae25db2035e600bfb37635407778533742ba04cfac11b3bc1413cc8dc6 not found: ID does not exist"
Jul 01 19:03:22.023777 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.023766    2583 scope.go:117] "RemoveContainer" containerID="d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"
Jul 01 19:03:22.023987 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:22.023972    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5\": container with ID starting with d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5 not found: ID does not exist" containerID="d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"
Jul 01 19:03:22.024024 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.023993    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5"} err="failed to get container status \"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5\": rpc error: code = NotFound desc = could not find container \"d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5\": container with ID starting with d5668d7e63f08c80259e380e8935beae2e4f40e742dd3ea078ce6ec7bccc3fe5 not found: ID does not exist"
Jul 01 19:03:22.365076 ip-10-0-137-241 kubenswrapper[2583]: E0701 19:03:22.365052    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hddln\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m67ws\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" pod="openshift-must-gather-m67ws/must-gather-hddln"
Jul 01 19:03:22.366096 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:22.366074    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" path="/var/lib/kubelet/pods/abf578bf-a994-4e7e-923e-ed3ab5e7e08d/volumes"
Jul 01 19:03:23.387679 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.387651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/alertmanager/0.log"
Jul 01 19:03:23.416563 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.416513    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/config-reloader/0.log"
Jul 01 19:03:23.444403 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.444371    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/kube-rbac-proxy-web/0.log"
Jul 01 19:03:23.472013 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.471986    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/kube-rbac-proxy/0.log"
Jul 01 19:03:23.504595 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.504575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/kube-rbac-proxy-metric/0.log"
Jul 01 19:03:23.532026 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.532004    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/prom-label-proxy/0.log"
Jul 01 19:03:23.564329 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.564310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3af68a2-cc04-4c8d-a79d-386e00e2e130/init-config-reloader/0.log"
Jul 01 19:03:23.618743 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.618718    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-9vsdc_4e91f2e0-0cf9-44d1-b2cb-f915acb5a399/cluster-monitoring-operator/0.log"
Jul 01 19:03:23.657577 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.657526    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8cb98_ab6700df-741e-4531-855d-d09717ed35d8/kube-state-metrics/0.log"
Jul 01 19:03:23.686524 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.686501    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8cb98_ab6700df-741e-4531-855d-d09717ed35d8/kube-rbac-proxy-main/0.log"
Jul 01 19:03:23.716471 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.716446    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8cb98_ab6700df-741e-4531-855d-d09717ed35d8/kube-rbac-proxy-self/0.log"
Jul 01 19:03:23.891933 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.891914    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpcvc_2a7be955-7677-4339-8d73-7ced4799a754/node-exporter/0.log"
Jul 01 19:03:23.919876 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.919819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpcvc_2a7be955-7677-4339-8d73-7ced4799a754/kube-rbac-proxy/0.log"
Jul 01 19:03:23.945765 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:23.945748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpcvc_2a7be955-7677-4339-8d73-7ced4799a754/init-textfile/0.log"
Jul 01 19:03:24.174937 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.174875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-zhm97_7f4b3d00-e5c7-4864-be13-9e696c4952aa/kube-rbac-proxy-main/0.log"
Jul 01 19:03:24.210900 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.210868    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-zhm97_7f4b3d00-e5c7-4864-be13-9e696c4952aa/kube-rbac-proxy-self/0.log"
Jul 01 19:03:24.236666 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.236647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-zhm97_7f4b3d00-e5c7-4864-be13-9e696c4952aa/openshift-state-metrics/0.log"
Jul 01 19:03:24.288114 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.288093    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/prometheus/0.log"
Jul 01 19:03:24.322581 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.322562    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/config-reloader/0.log"
Jul 01 19:03:24.350991 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.350972    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/thanos-sidecar/0.log"
Jul 01 19:03:24.379380 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.379365    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/kube-rbac-proxy-web/0.log"
Jul 01 19:03:24.409890 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.409867    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/kube-rbac-proxy/0.log"
Jul 01 19:03:24.443627 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.443554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/kube-rbac-proxy-thanos/0.log"
Jul 01 19:03:24.473077 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.473024    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_602fb654-5a74-4edc-8044-e1077c3c558c/init-config-reloader/0.log"
Jul 01 19:03:24.505755 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.505738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-v5ccj_4e74f6c8-ceae-4652-8bc8-9e84089d17ed/prometheus-operator/0.log"
Jul 01 19:03:24.528597 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.528583    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-v5ccj_4e74f6c8-ceae-4652-8bc8-9e84089d17ed/kube-rbac-proxy/0.log"
Jul 01 19:03:24.556863 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:24.556841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-52lg5_891e41bf-d175-4dcd-9c68-8bbef7bfa9c8/prometheus-operator-admission-webhook/0.log"
Jul 01 19:03:26.259474 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:26.259384    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-hswzv_73a72b23-be7c-4fac-8e30-5d5862adde93/networking-console-plugin/0.log"
Jul 01 19:03:26.999773 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:26.999744    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"]
Jul 01 19:03:27.000088 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="gather"
Jul 01 19:03:27.000146 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000090    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="gather"
Jul 01 19:03:27.000146 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000098    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="copy"
Jul 01 19:03:27.000146 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000103    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="copy"
Jul 01 19:03:27.000243 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000155    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="gather"
Jul 01 19:03:27.000243 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.000163    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf578bf-a994-4e7e-923e-ed3ab5e7e08d" containerName="copy"
Jul 01 19:03:27.002458 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.002441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.004989 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.004960    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s84nm\"/\"kube-root-ca.crt\""
Jul 01 19:03:27.004989 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.004985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s84nm\"/\"default-dockercfg-tmflf\""
Jul 01 19:03:27.005686 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.005672    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s84nm\"/\"openshift-service-ca.crt\""
Jul 01 19:03:27.015085 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.015065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"]
Jul 01 19:03:27.062108 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.062081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7dh\" (UniqueName: \"kubernetes.io/projected/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-kube-api-access-6c7dh\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.062222 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.062117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-lib-modules\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.062222 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.062159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-podres\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.062222 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.062190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-sys\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.062391 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.062245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-proc\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163397 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-podres\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163492 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-sys\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163492 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-proc\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163492 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7dh\" (UniqueName: \"kubernetes.io/projected/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-kube-api-access-6c7dh\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163492 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-lib-modules\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163694 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163508    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-sys\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163694 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163515    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-proc\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163694 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-podres\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.163694 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.163586    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-lib-modules\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.174925 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.174902    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7dh\" (UniqueName: \"kubernetes.io/projected/2f6054c2-af6a-4d6d-ba6c-aac848175d5c-kube-api-access-6c7dh\") pod \"perf-node-gather-daemonset-dzt6x\" (UID: \"2f6054c2-af6a-4d6d-ba6c-aac848175d5c\") " pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.311857 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.311800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:27.435934 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:27.435828    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"]
Jul 01 19:03:27.438594 ip-10-0-137-241 kubenswrapper[2583]: W0701 19:03:27.438561    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f6054c2_af6a_4d6d_ba6c_aac848175d5c.slice/crio-a8603ac3a241d568cbae6c96a55c9d4bd339f3b83d82cc655c69a0c9b4be8039 WatchSource:0}: Error finding container a8603ac3a241d568cbae6c96a55c9d4bd339f3b83d82cc655c69a0c9b4be8039: Status 404 returned error can't find the container with id a8603ac3a241d568cbae6c96a55c9d4bd339f3b83d82cc655c69a0c9b4be8039
Jul 01 19:03:28.025081 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.025050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x" event={"ID":"2f6054c2-af6a-4d6d-ba6c-aac848175d5c","Type":"ContainerStarted","Data":"cd7f07d024fa83efd2fe2f66050b2cef5289cfcb0cfe383e6ab2cc11e22ab133"}
Jul 01 19:03:28.025245 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.025095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x" event={"ID":"2f6054c2-af6a-4d6d-ba6c-aac848175d5c","Type":"ContainerStarted","Data":"a8603ac3a241d568cbae6c96a55c9d4bd339f3b83d82cc655c69a0c9b4be8039"}
Jul 01 19:03:28.025245 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.025163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:28.043751 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.043709    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x" podStartSLOduration=2.043696981 podStartE2EDuration="2.043696981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:03:28.042499638 +0000 UTC m=+1566.058079183" watchObservedRunningTime="2026-07-01 19:03:28.043696981 +0000 UTC m=+1566.059276525"
Jul 01 19:03:28.798395 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.798366    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s4lpv_7fae58e6-d31b-4218-a13e-826368c57ed1/dns/0.log"
Jul 01 19:03:28.820936 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.820910    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s4lpv_7fae58e6-d31b-4218-a13e-826368c57ed1/kube-rbac-proxy/0.log"
Jul 01 19:03:28.908986 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:28.908957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64k9d_0d52c436-737c-49d3-bf32-a3c80dec20e1/dns-node-resolver/0.log"
Jul 01 19:03:29.551460 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:29.551430    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mrwgc_ef1f2162-8c0c-42df-87fb-ae2da76b0cd0/node-ca/0.log"
Jul 01 19:03:30.798709 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:30.798672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-24wpt_0c3aa96c-ede2-4d93-a113-a362c3820669/serve-healthcheck-canary/0.log"
Jul 01 19:03:31.451501 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:31.451470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzctv_e916a9c4-a7ad-458b-8250-343d315cb106/kube-rbac-proxy/0.log"
Jul 01 19:03:31.475320 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:31.475274    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzctv_e916a9c4-a7ad-458b-8250-343d315cb106/exporter/0.log"
Jul 01 19:03:31.501371 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:31.501350    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzctv_e916a9c4-a7ad-458b-8250-343d315cb106/extractor/0.log"
Jul 01 19:03:33.786994 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:33.786966    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-qhjgv_af7a97bd-880e-43ce-8e15-102d8f6df7a2/manager/0.log"
Jul 01 19:03:33.828717 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:33.828688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9sg8p_197ec711-2289-4602-bda4-db0875c6bdcb/s3-init/0.log"
Jul 01 19:03:33.848388 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:33.848368    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9sg8p_197ec711-2289-4602-bda4-db0875c6bdcb/download-hf-model/0.log"
Jul 01 19:03:34.038199 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:34.038144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s84nm/perf-node-gather-daemonset-dzt6x"
Jul 01 19:03:39.733849 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.733823    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:03:39.758317 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.758275    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/egress-router-binary-copy/0.log"
Jul 01 19:03:39.780193 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.780166    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/cni-plugins/0.log"
Jul 01 19:03:39.806988 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.806968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/bond-cni-plugin/0.log"
Jul 01 19:03:39.832835 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.832813    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/routeoverride-cni/0.log"
Jul 01 19:03:39.856758 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.856741    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/whereabouts-cni-bincopy/0.log"
Jul 01 19:03:39.879820 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:39.879802    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mg9d_8a63e83e-5bc8-417d-9d12-ccb369d53076/whereabouts-cni/0.log"
Jul 01 19:03:40.378605 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:40.378575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-szrhw_113e4450-50df-42f3-b6d5-44b62ed68b61/kube-multus/0.log"
Jul 01 19:03:40.406929 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:40.406905    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4pb8p_03bffaf8-f1aa-48f2-a03f-e2161a1e7e86/network-metrics-daemon/0.log"
Jul 01 19:03:40.432440 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:40.432421    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4pb8p_03bffaf8-f1aa-48f2-a03f-e2161a1e7e86/kube-rbac-proxy/0.log"
Jul 01 19:03:41.366207 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.366174    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-controller/0.log"
Jul 01 19:03:41.385948 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.385928    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/1.log"
Jul 01 19:03:41.386109 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.385962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovn-acl-logging/0.log"
Jul 01 19:03:41.408613 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.408586    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/kube-rbac-proxy-node/0.log"
Jul 01 19:03:41.432858 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.432836    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:03:41.455798 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.455778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/northd/0.log"
Jul 01 19:03:41.482790 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.482766    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/nbdb/0.log"
Jul 01 19:03:41.508061 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.508046    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/sbdb/0.log"
Jul 01 19:03:41.580011 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:41.579989    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bpxvb_18b8b9ab-f5d2-4f95-9be8-a0237c6c9533/ovnkube-controller/0.log"
Jul 01 19:03:43.134498 ip-10-0-137-241 kubenswrapper[2583]: I0701 19:03:43.134464    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2kb8s_4fd1744c-8214-49ea-b977-4dfb8f751960/network-check-target-container/0.log"