Jul 02 13:12:32.430510 ip-10-0-141-149 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 13:12:32.430521 ip-10-0-141-149 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 13:12:32.430528 ip-10-0-141-149 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 13:12:32.430735 ip-10-0-141-149 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 13:12:42.431490 ip-10-0-141-149 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 13:12:42.431513 ip-10-0-141-149 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 33aae2fa0db34ec1ae9a7406ce6569c8 --
Jul 02 13:14:48.004135 ip-10-0-141-149 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.428433    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435042    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435054    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435057    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435061    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435064    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435066    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435069    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.448224 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435072    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435074    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435077    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435079    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435083    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435086    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435088    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435091    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435094    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435097    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435099    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435102    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435116    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435121    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435125    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435128    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435131    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435133    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435136    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435139    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.449649 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435142    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435145    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435147    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435150    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435153    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435157    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435159    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435162    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435164    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435167    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435169    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435172    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435174    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435177    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435180    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435183    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435186    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435189    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435192    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435196    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.450348 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435198    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435201    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435203    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435206    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435208    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435211    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435214    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435216    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435218    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435221    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435224    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435226    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435228    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435231    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435234    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435237    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435239    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435242    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435244    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.456360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435247    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435249    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435251    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435254    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435256    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435259    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435262    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435264    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435266    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435269    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435271    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435274    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435278    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435280    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435286    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435289    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435292    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435295    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435298    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.457462 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435300    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435303    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435305    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435308    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435311    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435315    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435319    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435322    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435325    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435327    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435330    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435332    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435335    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.435338    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437064    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437074    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437077    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437081    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437084    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.459382 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437087    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437090    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437092    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437095    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437099    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437102    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437104    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437107    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437110    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437113    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437115    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437118    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437122    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437126    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437128    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437131    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437134    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437137    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437139    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437142    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.460119 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437144    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437148    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437152    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437155    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437158    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437161    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437164    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437166    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437169    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437171    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437174    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437176    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437179    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437181    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437184    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437186    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437189    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437191    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437194    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437196    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.461167 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437199    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437202    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437207    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437209    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437212    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437215    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437217    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437222    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437225    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437229    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437232    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437235    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437238    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437240    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437243    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437246    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437249    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437251    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437255    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.467473 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437257    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437260    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437262    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437265    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437267    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437270    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437273    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437275    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437277    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437280    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437282    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437286    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437288    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437291    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437293    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437297    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437300    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437302    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437305    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.468239 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437307    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437311    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437314    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437317    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437319    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437321    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437324    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437326    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437329    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437331    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437334    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437336    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437338    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437341    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437343    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437346    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437708    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437713    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437716    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437719    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.469083 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437722    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437724    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437727    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437730    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437736    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437739    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437742    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437744    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437748    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437751    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437753    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437756    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437758    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437763    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437766    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437769    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437771    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437774    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437776    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437778    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.471701 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437781    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437784    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437786    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437789    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437791    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437794    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437796    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437799    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437801    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437804    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437806    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437809    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437812    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437814    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437816    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437820    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437822    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437825    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437828    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437830    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.570566 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437834    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.509819 ip-10-0-141-149 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437836    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437839    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437841    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437844    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437847    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437852    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437855    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437858    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437862    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437864    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437867    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437886    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437889    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437893    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437896    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437898    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437901    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437903    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.571999 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437906    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437908    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437912    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437916    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437919    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437922    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437924    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437927    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437929    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437932    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437934    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437938    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437940    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437945    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437947    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437949    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437952    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437954    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437958    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.575197 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437960    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437963    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437965    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437967    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437970    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437972    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437975    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437978    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437980    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437983    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437985    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437988    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437990    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437993    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437995    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.437998    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438001    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438074    2552 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438081    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438087    2552 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438091    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:14:48.575950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438095    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438098    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438102    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438109    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438114    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438118    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438122    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438125    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438128    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438131    2552 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438135    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438138    2552 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438141    2552 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438144    2552 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438148    2552 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438151    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438154    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438157    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438160    2552 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438164    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438166    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438169    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438173    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438176    2552 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:14:48.577860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438179    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438182    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438185    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438188    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438192    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438195    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438198    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438201    2552 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438204    2552 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438207    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438211    2552 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438214    2552 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438216    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438222    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438225    2552 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438228    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438231    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438234    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438238    2552 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438241    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438244    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438246    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438250    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438253    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438255    2552 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:14:48.579032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438258    2552 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438262    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438265    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438268    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438271    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438273    2552 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438276    2552 flags.go:64] FLAG: --help="false"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438279    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-141-149.ec2.internal"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438282    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438285    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438288    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438291    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438294    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438297    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438300    2552 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438303    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438305    2552 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438308    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438311    2552 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438314    2552 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438318    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438322    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438325    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438327    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:14:48.581372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438330    2552 flags.go:64] FLAG: --lock-file=""
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438334    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438337    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438340    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438345    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438348    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438351    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438354    2552 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438356    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438360    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438363    2552 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438365    2552 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438369    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438372    2552 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438376    2552 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438379    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438382    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438385    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438388    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438391    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438394    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438397    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438403    2552 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438406    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438409    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:14:48.582470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438411    2552 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438414    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438419    2552 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438424    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438427    2552 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438430    2552 flags.go:64] FLAG: --port="10250"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438433    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438436    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086e39c012c5f45bd"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438439    2552 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438444    2552 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438447    2552 flags.go:64] FLAG: --register-node="true"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438450    2552 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438454    2552 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438457    2552 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438459    2552 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438462    2552 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438465    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438468    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438471    2552 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438474    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438476    2552 flags.go:64] FLAG: --runonce="false"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438479    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438482    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438485    2552 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438488    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:14:48.583243 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438490    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438493    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438496    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438499    2552 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438502    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438505    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438508    2552 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438510    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438513    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438516    2552 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438520    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438525    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438528    2552 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438531    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438535    2552 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438538    2552 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438543    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438546    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438549    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438552    2552 flags.go:64] FLAG: --v="2"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438556    2552 flags.go:64] FLAG: --version="false"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438559    2552 flags.go:64] FLAG: --vmodule=""
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438563    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.438566    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438677    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.584402 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438680    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438683    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438687    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438689    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438692    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438694    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438697    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438699    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438702    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438704    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438707    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438709    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438711    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438714    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438716    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438719    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438721    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438726    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438729    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438731    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.585176 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438733    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438736    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438739    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438742    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438745    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438748    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438750    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438752    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438755    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438757    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438760    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438763    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438766    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438768    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438771    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438773    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438776    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438778    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438780    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438783    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.585883 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438785    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438789    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438793    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438795    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438798    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438801    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438804    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438807    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438809    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438813    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438817    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438820    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438823    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438826    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438829    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438833    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438836    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438839    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438842    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.586656 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438845    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438848    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438851    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438853    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438856    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438859    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438862    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438865    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438867    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438885    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438889    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438892    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438894    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438897    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438899    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438902    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438904    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438907    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438909    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438912    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.587418 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438915    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438917    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438921    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438924    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438926    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438929    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438931    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438934    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438938    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438941    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438945    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438948    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438951    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438954    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438956    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438959    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438961    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438964    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.588205 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.438967    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.439503    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.445654    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.445668    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.445684    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.445693    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445753    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445758    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445762    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445765    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445767    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445771    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445775    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445779    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.588989 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445782    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445785    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445787    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445790    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445793    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445797    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445801    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445805    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445809    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445811    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445815    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445817    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445820    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445822    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445825    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445827    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445830    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445832    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445835    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445838    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.589753 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445841    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445844    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445847    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445849    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445852    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445855    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445857    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445860    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445862    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445865    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445867    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445883    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445887    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445891    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445893    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445895    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445899    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445901    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445904    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445906    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.590537 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445909    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445912    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445918    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445922    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445925    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445928    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445931    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445934    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445937    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445940    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445943    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445946    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445948    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445951    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445956    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445961    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445965    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445968    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445970    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445973    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.592480 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445976    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445978    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445981    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445986    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445990    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445993    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445996    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.445999    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446002    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446004    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446007    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446009    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446012    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446014    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446018    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446021    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446023    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446026    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446030    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.593219 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446034    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446038    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446043    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446045    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446048    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446050    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446053    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446055    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446058    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446061    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446063    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446065    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.446070    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446183    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446191    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:48.595490 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446195    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446198    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446202    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446206    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446210    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446214    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446217    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446220    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446224    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446226    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446229    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446232    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446236    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446238    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446241    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446244    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446246    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446249    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446251    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:48.596256 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446254    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446256    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446258    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446262    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446266    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446270    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446274    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446277    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446280    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446282    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446285    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446287    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446290    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446292    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446295    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446299    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446302    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446305    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446307    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446310    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:48.689869 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446312    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446315    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446318    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446320    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446323    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446327    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446329    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446332    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446334    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446338    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446342    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446346    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446350    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446353    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446355    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446358    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446360    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446363    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446366    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:48.690928 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446370    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446373    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446376    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446380    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446382    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446385    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446388    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446391    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446395    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446397    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446400    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446402    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446405    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446407    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446410    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446412    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446416    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446420    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446426    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:48.691691 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446430    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446432    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446435    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446437    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446440    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446442    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446446    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446448    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446451    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446453    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446456    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446458    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446460    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446463    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446465    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446470    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446472    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446475    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446477    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:48.692456 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:48.446479    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.446484    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.446587    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.449584    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.450813    2552 server.go:1019] "Starting client certificate rotation"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.450902    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.450932    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.453384    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.460076    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.471382    2552 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.475307    2552 log.go:25] "Validated CRI v1 image API"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.477184    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:14:48.693209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.480146    2552 fs.go:135] Filesystem UUIDs: map[1246dfc5-4381-4628-80e7-9a41de397705:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e942b5a8-bae8-4a74-98f2-c04eb034eb3e:/dev/nvme0n1p4]
Jul 02 13:14:48.694727 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.480166    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.484845    2552 manager.go:216] Machine: {Timestamp:2026-07-02 13:14:48.483710501 +0000 UTC m=+0.339103396 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096825 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29007628d315f5db617621b8db4c66 SystemUUID:ec290076-28d3-15f5-db61-7621b8db4c66 BootID:33aae2fa-0db3-4ec1-ae9a-7406ce6569c8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0d:6d:3f:38:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0d:6d:3f:38:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:0a:8b:e4:7f:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.484961    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.485027    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.488478    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.488507    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.489738    2552 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.489750    2552 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:14:48.695126 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.489760    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.490597    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.491390    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.493852    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494338    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494521    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494563    2552 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494598    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494612    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494626    2552 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.494641    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.495221    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.495247    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.497946    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.499570    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.499600    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500140    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500154    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500160    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500165    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500173    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500179    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500349    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500355    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500362    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500368    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500377    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.500385    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.501073    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.501086    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.504533    2552 server.go:1268] "Started kubelet"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.510231    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.510286    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.511934    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.511996    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.512258    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.513850    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.515624    2552 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.517860    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.517995    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.518100    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.518765    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:14:48.695498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.518970    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.518984    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.519108    2552 factory.go:55] Registering systemd factory
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.519124    2552 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.519112    2552 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.519160    2552 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520020    2552 factory.go:153] Registering CRI-O factory
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520039    2552 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520094    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520124    2552 factory.go:103] Registering Raw factory
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520140    2552 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.520542    2552 manager.go:349] Starting recovery of all containers
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.526036    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.526266    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.526664    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.531428    2552 manager.go:354] Recovery completed
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.532024    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.532198    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-149.ec2.internal.18be7b509482f9fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-149.ec2.internal,UID:ip-10-0-141-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-149.ec2.internal,},FirstTimestamp:2026-07-02 13:14:48.504506878 +0000 UTC m=+0.359899756,LastTimestamp:2026-07-02 13:14:48.504506878 +0000 UTC m=+0.359899756,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-149.ec2.internal,}"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.535887    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.537933    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.537982    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.537997    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.538471    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.538480    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.538485    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.538503    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.540014    2552 policy_none.go:49] "None policy: Start"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.540026    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.540036    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.541003    2552 policy_none.go:47] "Start"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.616806    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5774"
Jul 02 13:14:48.697052 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.619000    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:48.698993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:48.635986    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5774"
Jul 02 13:14:48.805714 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.719065    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:48.820537 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.819754    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:48.921119 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:48.920126    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.035698 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.020979    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.153754 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.121531    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.273207 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.222126    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.374293 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.322854    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.479812 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.423917    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.479812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.450141    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:14:49.512096 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.512077    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.512205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.512112    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.512205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.512130    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.523747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.523703    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:49.523994 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.523975    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.524897 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.524882    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:49.530574 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.530562    2552 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:14:49.530639 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.530626    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:14:49.530678 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.530642    2552 server.go:87] "Starting device plugin registration server"
Jul 02 13:14:49.530945 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.530927    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:14:49.531055 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.530962    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:14:49.531115 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.531099    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:14:49.531204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.531180    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:14:49.531204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.531194    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:14:49.531636 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.531616    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:14:49.531716 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.531673    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.536730 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.536602    2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-149.ec2.internal\" not found" node="ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.536820 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.536733    2552 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-149.ec2.internal" not found
Jul 02 13:14:49.545888 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.545867    2552 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2584 into the system container \"/system.slice\": <nil>"
Jul 02 13:14:49.558761 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.558741    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.560010    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.560048    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.560068    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.560107    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.563102    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:49.564650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.563364    2552 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-149.ec2.internal" not found
Jul 02 13:14:49.592841 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.592823    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:49.630252 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.630212    2552 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-149.ec2.internal" not found
Jul 02 13:14:49.631356 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.631343    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.632210 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.632191    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.632308 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.632225    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.632308 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.632237    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.632308 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.632267    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.640066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.640033    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:09:49 +0000 UTC" deadline="2028-02-16 12:34:00.302830665 +0000 UTC"
Jul 02 13:14:49.640066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.640064    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14255h19m10.662769399s"
Jul 02 13:14:49.644637 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.644622    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.644688 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.644644    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-149.ec2.internal\": node \"ip-10-0-141-149.ec2.internal\" not found"
Jul 02 13:14:49.660701 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.660680    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"]
Jul 02 13:14:49.662781 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.662759    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.663840 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.663824    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.692617 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.692601    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.697331 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.697315    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.718228 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.718206    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.726079 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.726060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0695f0d6e8402fe098c22a36367da2e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-149.ec2.internal\" (UID: \"d0695f0d6e8402fe098c22a36367da2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.726145 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.726083    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.726145 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.726103    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.727058 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.727043    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:49.727361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.727349    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:49.774154 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.774135    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:49.774226 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.774166    2552 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.774226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.774176    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.807490 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.807472    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:49.807577 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:49.807510    2552 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-149.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826213 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826189    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826217    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0695f0d6e8402fe098c22a36367da2e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-149.ec2.internal\" (UID: \"d0695f0d6e8402fe098c22a36367da2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826354 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826290    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826354 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826325    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1294756ed9becdbbc18223f82bfcc14a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal\" (UID: \"1294756ed9becdbbc18223f82bfcc14a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:49.826354 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:49.826336    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0695f0d6e8402fe098c22a36367da2e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-149.ec2.internal\" (UID: \"d0695f0d6e8402fe098c22a36367da2e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:50.031341 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.031285    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:50.033250 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.033232    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal"
Jul 02 13:14:50.495762 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.495691    2552 apiserver.go:52] "Watching apiserver"
Jul 02 13:14:50.506380 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.506356    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:14:50.506741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.506719    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-q9mnj","openshift-network-diagnostics/network-check-target-w5957","kube-system/global-pull-secret-syncer-stzdv","kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gmjck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal","openshift-multus/multus-additional-cni-plugins-s89wm","openshift-multus/multus-rlnbl","openshift-network-operator/iptables-alerter-g8qvd","openshift-ovn-kubernetes/ovnkube-node-nkxrg","kube-system/konnectivity-agent-tvlpw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7","openshift-dns/node-resolver-546hs","openshift-image-registry/node-ca-r8wzf"]
Jul 02 13:14:50.508541 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.508524    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.508627 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.508606    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:14:50.509717 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.509702    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:50.509775 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.509758    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:14:50.510724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.510708    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.510783 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.510767    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:14:50.511764 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.511750    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.512974 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.512956    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.514234 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.514212    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.515479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.515463    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.517053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.517035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.517282 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.517263    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:14:50.517374 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.517312    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cg7r5\""
Jul 02 13:14:50.518224 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.518205    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.518342 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.518323    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:14:50.518671 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.518654    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.519169 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519146    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:50.519553 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519535    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.519801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519773    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.519907 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519805    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.519907 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519836    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.520031 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.519948    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.520825 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.520805    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.520825 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.520823    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:14:50.520969 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.520940    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:14:50.521303 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521096    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7s6pn\""
Jul 02 13:14:50.521303 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521200    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:14:50.521303 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521242    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:14:50.521303 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521266    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.521528 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521409    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:14:50.521528 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521434    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rkz5b\""
Jul 02 13:14:50.521792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521770    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:14:50.521892 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.521869    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.522073 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522055    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mxrx9\""
Jul 02 13:14:50.522300 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522283    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:14:50.522434 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522413    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:14:50.522612 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522587    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l6rts\""
Jul 02 13:14:50.522700 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522651    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pmk6z\""
Jul 02 13:14:50.522700 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522665    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:14:50.522860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.522841    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.523231 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.523214    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.523966 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.523942    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c2f9m\""
Jul 02 13:14:50.527987 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.527458    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.529012 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.528966    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-system-cni-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.529100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529043    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-socket-dir-parent\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.529144 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529121    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-multus\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.529212 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529169    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-daemon-config\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.529392 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-binary-copy\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.529437 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529414    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-systemd-units\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.529498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529441    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-var-lib-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.529551 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-script-lib\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.529605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529569    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjz5\" (UniqueName: \"kubernetes.io/projected/3590789f-cff0-43ff-8158-e0f5ab08ea49-kube-api-access-jqjz5\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.529657 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529628    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4vq\" (UniqueName: \"kubernetes.io/projected/a62feb9f-c762-41b5-804b-7abe8c0facac-kube-api-access-jh4vq\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.529706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529678    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.529756 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529709    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8cv\" (UniqueName: \"kubernetes.io/projected/c86e3245-2705-4a72-b608-61bebd9eb17e-kube-api-access-zb8cv\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.529756 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529740    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.529850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529769    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-run\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.529850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529799    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-bin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.529850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.530015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-cnibin\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.530015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.529919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-netns\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.530114 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-ovn\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.530167 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530137    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-registration-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.530222 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-var-lib-kubelet\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.530330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530252    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-os-release\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.530330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530282    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-log-socket\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.530330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530290    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.530480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530340    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-socket-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.530480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530395    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-tuned\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.530578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530455    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.530578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-dbus\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.530578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530570    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-modprobe-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.530724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530605    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-systemd\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.530724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530673    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.530724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530715    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-os-release\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.530878 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530746    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-k8s-cni-cncf-io\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.530878 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530809    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-node-log\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.530878 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530842    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-bin\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.531055 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.530997    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-kubelet-config\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.531055 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531043    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.531175 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysconfig\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.531226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531195    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.531511 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531490    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:14:50.531589 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531195    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-conf\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.531652 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531609    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-host\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.531652 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531648    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-tmp\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.531758 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531682    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/354e948f-dc63-416a-8642-8598553c6185-konnectivity-ca\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.531758 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531694    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmvbx\""
Jul 02 13:14:50.531758 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531568    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6wfdt\""
Jul 02 13:14:50.531900 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531826    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:14:50.531900 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531815    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e636dd3e-e640-47e6-8d2f-b16930340638-ovn-node-metrics-cert\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.532014 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531907    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:14:50.532014 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.531916    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:50.532014 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532005    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-kubernetes\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.532161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532087    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-conf-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.532216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532195    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8n9h\" (UniqueName: \"kubernetes.io/projected/37e92131-f21d-4309-9f51-9215d2221123-kube-api-access-v8n9h\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.532251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532147    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:14:50.532251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-systemd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.532388 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkgrf\" (UniqueName: \"kubernetes.io/projected/497e17e1-0fd9-467d-bf51-b261ffe73571-kube-api-access-dkgrf\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.532726 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532708    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-hostroot\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.532773 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532758    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e92131-f21d-4309-9f51-9215d2221123-host-slash\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.532809 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532796    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.532854 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532835    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-etc-kubernetes\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.532900 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532869    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49mx\" (UniqueName: \"kubernetes.io/projected/e636dd3e-e640-47e6-8d2f-b16930340638-kube-api-access-p49mx\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533019 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532934    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-sys-fs\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.533019 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532973    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-lib-modules\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.533019 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.532999    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.533160 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533022    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cni-binary-copy\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533160 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533047    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-multus-certs\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533160 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533092    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/354e948f-dc63-416a-8642-8598553c6185-agent-certs\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.533160 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533128    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-etc-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533161    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-sys\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cnibin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-kubelet\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533238    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-kubelet\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533262    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-netns\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533287    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-netd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533322    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-etc-selinux\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.533360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533372    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-system-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m664m\" (UniqueName: \"kubernetes.io/projected/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-kube-api-access-m664m\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533419    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-slash\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533452    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533477    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-env-overrides\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-device-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533530    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37e92131-f21d-4309-9f51-9215d2221123-iptables-alerter-script\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533564    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533588    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-config\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.533732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.533619    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.540270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.540243    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:50.571801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.571784    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8mbh"
Jul 02 13:14:50.584091 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.584074    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8mbh"
Jul 02 13:14:50.620066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.620048    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:14:50.634102 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634076    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-sys\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.634261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634117    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cnibin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634149    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-kubelet\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634179    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-kubelet\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-netns\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634247    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-netd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634290    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-etc-selinux\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634318    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634350    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-system-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634385    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m664m\" (UniqueName: \"kubernetes.io/projected/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-kube-api-access-m664m\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634412    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-slash\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634446    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-env-overrides\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634510 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634504    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-device-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckk62\" (UniqueName: \"kubernetes.io/projected/1cb9273a-103d-4f41-b823-1a2c4f6029d5-kube-api-access-ckk62\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634574    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37e92131-f21d-4309-9f51-9215d2221123-iptables-alerter-script\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634602    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634633    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-config\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634669    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-etc-selinux\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634690    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27sr7\" (UniqueName: \"kubernetes.io/projected/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-kube-api-access-27sr7\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634723    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-system-cni-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634754    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-socket-dir-parent\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634752    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cnibin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-multus\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634815    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-daemon-config\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634818    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-kubelet\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.634889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634843    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-binary-copy\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634870    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-kubelet\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-systemd-units\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-var-lib-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634926    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-systemd-units\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.634977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-script-lib\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635009    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cb9273a-103d-4f41-b823-1a2c4f6029d5-hosts-file\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635049    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjz5\" (UniqueName: \"kubernetes.io/projected/3590789f-cff0-43ff-8158-e0f5ab08ea49-kube-api-access-jqjz5\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.635067    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4vq\" (UniqueName: \"kubernetes.io/projected/a62feb9f-c762-41b5-804b-7abe8c0facac-kube-api-access-jh4vq\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb9273a-103d-4f41-b823-1a2c4f6029d5-tmp-dir\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.635187    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.135155064 +0000 UTC m=+2.990547954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635218    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8cv\" (UniqueName: \"kubernetes.io/projected/c86e3245-2705-4a72-b608-61bebd9eb17e-kube-api-access-zb8cv\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.635564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635264    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.635288    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-run\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635324    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-bin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635357    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635391    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-netns\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635450    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-cnibin\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635459    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-netd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-netns\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635497    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-var-lib-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-ovn\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635545    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-registration-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635576    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-var-lib-kubelet\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635602    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-os-release\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635643    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-log-socket\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.636289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635697    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-socket-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-tuned\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635782    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-dbus\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635813    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-modprobe-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635844    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-systemd\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635877    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.635911    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-os-release\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-k8s-cni-cncf-io\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636049    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-node-log\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636101    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-bin\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636134    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-kubelet-config\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysconfig\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636211    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-conf\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637077 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636231    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-script-lib\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-host\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636273    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-tmp\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636321    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/354e948f-dc63-416a-8642-8598553c6185-konnectivity-ca\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636354    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e636dd3e-e640-47e6-8d2f-b16930340638-ovn-node-metrics-cert\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636384    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-daemon-config\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636412    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-kubernetes\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636442    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-conf-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636497    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-conf-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636612    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636784    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37e92131-f21d-4309-9f51-9215d2221123-iptables-alerter-script\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636864    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-system-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.636938    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-ovnkube-config\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637013    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8n9h\" (UniqueName: \"kubernetes.io/projected/37e92131-f21d-4309-9f51-9215d2221123-kube-api-access-v8n9h\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.637778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637048    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-systemd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637097    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-systemd\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637101    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkgrf\" (UniqueName: \"kubernetes.io/projected/497e17e1-0fd9-467d-bf51-b261ffe73571-kube-api-access-dkgrf\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-serviceca\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637173    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-sys\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637180    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-hostroot\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637184    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-slash\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e92131-f21d-4309-9f51-9215d2221123-host-slash\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-host\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637239    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637265    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637291    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-socket-dir-parent\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-etc-kubernetes\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637325    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-device-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637333    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-etc-kubernetes\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.638790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637337    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49mx\" (UniqueName: \"kubernetes.io/projected/e636dd3e-e640-47e6-8d2f-b16930340638-kube-api-access-p49mx\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637367    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-sys-fs\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637382    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637414    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-sys-fs\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637416    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-lib-modules\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637470    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-multus-cni-dir\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637489    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cni-binary-copy\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637521    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-multus-certs\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637551    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/354e948f-dc63-416a-8642-8598553c6185-agent-certs\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637728    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-etc-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.637860    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-etc-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638073    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e636dd3e-e640-47e6-8d2f-b16930340638-env-overrides\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638175    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-run\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638209    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-ovn\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.639595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638226    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-netns\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638287    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-registration-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638331    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e92131-f21d-4309-9f51-9215d2221123-host-slash\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638378    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-hostroot\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638396    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.638447    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.138425875 +0000 UTC m=+2.993818756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638450    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-kubernetes\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638409    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-var-lib-kubelet\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638501    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-cnibin\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638509    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-run-openvswitch\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638621    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-log-socket\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.638719    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/497e17e1-0fd9-467d-bf51-b261ffe73571-socket-dir\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639234    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639566    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.640442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639702    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-lib-modules\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639795    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a62feb9f-c762-41b5-804b-7abe8c0facac-cni-binary-copy\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639890    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-os-release\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.639987    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-multus\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640234    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-dbus\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640276    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a62feb9f-c762-41b5-804b-7abe8c0facac-system-cni-dir\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640301    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-run-ovn-kubernetes\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640357    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-k8s-cni-cncf-io\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640373    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-os-release\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640392    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-host-cni-bin\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640450    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysconfig\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640455    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e636dd3e-e640-47e6-8d2f-b16930340638-node-log\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640508    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f6ceabc-fa61-470e-8df7-db311b0433f6-kubelet-config\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640545    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640560    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-var-lib-cni-bin\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640644    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-modprobe-d\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640747    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-systemd\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640863    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-sysctl-conf\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.640911    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-host-run-multus-certs\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641019    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-tmp\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641102    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-cni-binary-copy\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641104    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e636dd3e-e640-47e6-8d2f-b16930340638-ovn-node-metrics-cert\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641121    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3590789f-cff0-43ff-8158-e0f5ab08ea49-host\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641298    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3590789f-cff0-43ff-8158-e0f5ab08ea49-etc-tuned\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.641792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.641541    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/354e948f-dc63-416a-8642-8598553c6185-konnectivity-ca\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.643999 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.643973    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/354e948f-dc63-416a-8642-8598553c6185-agent-certs\") pod \"konnectivity-agent-tvlpw\" (UID: \"354e948f-dc63-416a-8642-8598553c6185\") " pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.650118 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.650088    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8cv\" (UniqueName: \"kubernetes.io/projected/c86e3245-2705-4a72-b608-61bebd9eb17e-kube-api-access-zb8cv\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:50.650419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.650396    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m664m\" (UniqueName: \"kubernetes.io/projected/5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e-kube-api-access-m664m\") pod \"multus-rlnbl\" (UID: \"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e\") " pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.651559 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.651539    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjz5\" (UniqueName: \"kubernetes.io/projected/3590789f-cff0-43ff-8158-e0f5ab08ea49-kube-api-access-jqjz5\") pod \"tuned-gmjck\" (UID: \"3590789f-cff0-43ff-8158-e0f5ab08ea49\") " pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.655417 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.655379    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:50.655417 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.655408    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:50.655417 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.655424    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:50.655982 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:50.655478    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.155462026 +0000 UTC m=+3.010854891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:50.657892 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.657505    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49mx\" (UniqueName: \"kubernetes.io/projected/e636dd3e-e640-47e6-8d2f-b16930340638-kube-api-access-p49mx\") pod \"ovnkube-node-nkxrg\" (UID: \"e636dd3e-e640-47e6-8d2f-b16930340638\") " pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.659726 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.659703    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkgrf\" (UniqueName: \"kubernetes.io/projected/497e17e1-0fd9-467d-bf51-b261ffe73571-kube-api-access-dkgrf\") pod \"aws-ebs-csi-driver-node-5dxc7\" (UID: \"497e17e1-0fd9-467d-bf51-b261ffe73571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.659814 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.659792    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4vq\" (UniqueName: \"kubernetes.io/projected/a62feb9f-c762-41b5-804b-7abe8c0facac-kube-api-access-jh4vq\") pod \"multus-additional-cni-plugins-s89wm\" (UID: \"a62feb9f-c762-41b5-804b-7abe8c0facac\") " pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.660171 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.660151    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8n9h\" (UniqueName: \"kubernetes.io/projected/37e92131-f21d-4309-9f51-9215d2221123-kube-api-access-v8n9h\") pod \"iptables-alerter-g8qvd\" (UID: \"37e92131-f21d-4309-9f51-9215d2221123\") " pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.726434 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.726399    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0695f0d6e8402fe098c22a36367da2e.slice/crio-8b09052483fc9b0ae4925286cbf4ec6f75d87fc160915a96dd48ec973b4550d7 WatchSource:0}: Error finding container 8b09052483fc9b0ae4925286cbf4ec6f75d87fc160915a96dd48ec973b4550d7: Status 404 returned error can't find the container with id 8b09052483fc9b0ae4925286cbf4ec6f75d87fc160915a96dd48ec973b4550d7
Jul 02 13:14:50.726932 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.726912    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1294756ed9becdbbc18223f82bfcc14a.slice/crio-60fa438d6e442011352e6f37593c6dd9c968fe92c4203653560864f5f9fd2431 WatchSource:0}: Error finding container 60fa438d6e442011352e6f37593c6dd9c968fe92c4203653560864f5f9fd2431: Status 404 returned error can't find the container with id 60fa438d6e442011352e6f37593c6dd9c968fe92c4203653560864f5f9fd2431
Jul 02 13:14:50.736013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.735999    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:14:50.738915 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.738879    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-serviceca\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.739038 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.738928    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-host\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.739100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739051    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckk62\" (UniqueName: \"kubernetes.io/projected/1cb9273a-103d-4f41-b823-1a2c4f6029d5-kube-api-access-ckk62\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.739170 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739097    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sr7\" (UniqueName: \"kubernetes.io/projected/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-kube-api-access-27sr7\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.739170 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739129    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cb9273a-103d-4f41-b823-1a2c4f6029d5-hosts-file\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.739170 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739162    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb9273a-103d-4f41-b823-1a2c4f6029d5-tmp-dir\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.739320 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739056    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-host\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.739320 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739255    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cb9273a-103d-4f41-b823-1a2c4f6029d5-hosts-file\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.739438 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739406    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-serviceca\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.739503 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.739442    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb9273a-103d-4f41-b823-1a2c4f6029d5-tmp-dir\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.752531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.752479    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27sr7\" (UniqueName: \"kubernetes.io/projected/a04c8f06-90e1-4c77-834b-e9d44cf8bfac-kube-api-access-27sr7\") pod \"node-ca-r8wzf\" (UID: \"a04c8f06-90e1-4c77-834b-e9d44cf8bfac\") " pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.752621 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.752603    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckk62\" (UniqueName: \"kubernetes.io/projected/1cb9273a-103d-4f41-b823-1a2c4f6029d5-kube-api-access-ckk62\") pod \"node-resolver-546hs\" (UID: \"1cb9273a-103d-4f41-b823-1a2c4f6029d5\") " pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.840571 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.840550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmjck"
Jul 02 13:14:50.847895 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.847860    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3590789f_cff0_43ff_8158_e0f5ab08ea49.slice/crio-e5f6243f90724a73a424dbcf5d10a38de0da088014e2aff02360c6235d0f7408 WatchSource:0}: Error finding container e5f6243f90724a73a424dbcf5d10a38de0da088014e2aff02360c6235d0f7408: Status 404 returned error can't find the container with id e5f6243f90724a73a424dbcf5d10a38de0da088014e2aff02360c6235d0f7408
Jul 02 13:14:50.848441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.848425    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s89wm"
Jul 02 13:14:50.855501 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.855478    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62feb9f_c762_41b5_804b_7abe8c0facac.slice/crio-6a09516ac35c29f1b3a05552347b0b439a0de4979127aafd27b2e10aabced2b7 WatchSource:0}: Error finding container 6a09516ac35c29f1b3a05552347b0b439a0de4979127aafd27b2e10aabced2b7: Status 404 returned error can't find the container with id 6a09516ac35c29f1b3a05552347b0b439a0de4979127aafd27b2e10aabced2b7
Jul 02 13:14:50.869719 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.869698    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlnbl"
Jul 02 13:14:50.876173 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.876147    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1fe7e5_d0c5_42fa_aec8_691835f4ae2e.slice/crio-7570d08ffdb9a71b5b4d9813cfcb9e1aafa286a7b6b9a0f95ee8e6cade586a02 WatchSource:0}: Error finding container 7570d08ffdb9a71b5b4d9813cfcb9e1aafa286a7b6b9a0f95ee8e6cade586a02: Status 404 returned error can't find the container with id 7570d08ffdb9a71b5b4d9813cfcb9e1aafa286a7b6b9a0f95ee8e6cade586a02
Jul 02 13:14:50.895959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.895936    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8qvd"
Jul 02 13:14:50.902360 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.902341    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e92131_f21d_4309_9f51_9215d2221123.slice/crio-3f864b512710e1d2bf49b410c038320a6681c74ae91fc3eb81e40a9628e1c00d WatchSource:0}: Error finding container 3f864b512710e1d2bf49b410c038320a6681c74ae91fc3eb81e40a9628e1c00d: Status 404 returned error can't find the container with id 3f864b512710e1d2bf49b410c038320a6681c74ae91fc3eb81e40a9628e1c00d
Jul 02 13:14:50.909080 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.909064    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:14:50.916003 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.915974    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode636dd3e_e640_47e6_8d2f_b16930340638.slice/crio-c9665c8ced5c59cb8ba888d24dcb89bba2329dfc435d8262883c0239361e7b26 WatchSource:0}: Error finding container c9665c8ced5c59cb8ba888d24dcb89bba2329dfc435d8262883c0239361e7b26: Status 404 returned error can't find the container with id c9665c8ced5c59cb8ba888d24dcb89bba2329dfc435d8262883c0239361e7b26
Jul 02 13:14:50.932909 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.932869    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:14:50.939714 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.939689    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354e948f_dc63_416a_8642_8598553c6185.slice/crio-afc9a8bfe0240b41052b77bb6b7e574ef38de07e456d2588cf6e2c044e88c72c WatchSource:0}: Error finding container afc9a8bfe0240b41052b77bb6b7e574ef38de07e456d2588cf6e2c044e88c72c: Status 404 returned error can't find the container with id afc9a8bfe0240b41052b77bb6b7e574ef38de07e456d2588cf6e2c044e88c72c
Jul 02 13:14:50.942653 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.942636    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7"
Jul 02 13:14:50.950081 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.950063    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497e17e1_0fd9_467d_bf51_b261ffe73571.slice/crio-e1263c8866710aff221aa2eede3e8ddc9ec97644b431e419fe2edf16232a2c0b WatchSource:0}: Error finding container e1263c8866710aff221aa2eede3e8ddc9ec97644b431e419fe2edf16232a2c0b: Status 404 returned error can't find the container with id e1263c8866710aff221aa2eede3e8ddc9ec97644b431e419fe2edf16232a2c0b
Jul 02 13:14:50.968084 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.968066    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-546hs"
Jul 02 13:14:50.973853 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:50.973835    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8wzf"
Jul 02 13:14:50.979407 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.979389    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb9273a_103d_4f41_b823_1a2c4f6029d5.slice/crio-0092ea93e50bd5f41708642387e18eaeba77b9a4ad4a6b528b2e35a0b1fdd511 WatchSource:0}: Error finding container 0092ea93e50bd5f41708642387e18eaeba77b9a4ad4a6b528b2e35a0b1fdd511: Status 404 returned error can't find the container with id 0092ea93e50bd5f41708642387e18eaeba77b9a4ad4a6b528b2e35a0b1fdd511
Jul 02 13:14:50.982332 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:14:50.982314    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04c8f06_90e1_4c77_834b_e9d44cf8bfac.slice/crio-e88b654df246dd80143617e5d9e45d9e01d6bca9f4ed38cfd5300e48d905acac WatchSource:0}: Error finding container e88b654df246dd80143617e5d9e45d9e01d6bca9f4ed38cfd5300e48d905acac: Status 404 returned error can't find the container with id e88b654df246dd80143617e5d9e45d9e01d6bca9f4ed38cfd5300e48d905acac
Jul 02 13:14:51.142858 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.142799    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:51.142962 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.142911    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.142962 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.142934    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:51.143038 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.143003    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.142984553 +0000 UTC m=+3.998377428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.143038 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.143014    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.143108 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.143058    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.14304361 +0000 UTC m=+3.998436473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.243678 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.243650    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:51.243812 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.243800    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:51.243880 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.243827    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:51.243880 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.243841    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:51.244002 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:51.243911    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.24388005 +0000 UTC m=+4.099272917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:51.581913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.581846    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8wzf" event={"ID":"a04c8f06-90e1-4c77-834b-e9d44cf8bfac","Type":"ContainerStarted","Data":"e88b654df246dd80143617e5d9e45d9e01d6bca9f4ed38cfd5300e48d905acac"}
Jul 02 13:14:51.586319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.586283    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:50 +0000 UTC" deadline="2028-03-02 06:14:11.103309979 +0000 UTC"
Jul 02 13:14:51.586319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.586321    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14608h59m19.516994176s"
Jul 02 13:14:51.596045 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.596012    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-546hs" event={"ID":"1cb9273a-103d-4f41-b823-1a2c4f6029d5","Type":"ContainerStarted","Data":"0092ea93e50bd5f41708642387e18eaeba77b9a4ad4a6b528b2e35a0b1fdd511"}
Jul 02 13:14:51.611767 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.611688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7" event={"ID":"497e17e1-0fd9-467d-bf51-b261ffe73571","Type":"ContainerStarted","Data":"e1263c8866710aff221aa2eede3e8ddc9ec97644b431e419fe2edf16232a2c0b"}
Jul 02 13:14:51.614557 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.614505    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvlpw" event={"ID":"354e948f-dc63-416a-8642-8598553c6185","Type":"ContainerStarted","Data":"afc9a8bfe0240b41052b77bb6b7e574ef38de07e456d2588cf6e2c044e88c72c"}
Jul 02 13:14:51.623423 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.623351    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlnbl" event={"ID":"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e","Type":"ContainerStarted","Data":"7570d08ffdb9a71b5b4d9813cfcb9e1aafa286a7b6b9a0f95ee8e6cade586a02"}
Jul 02 13:14:51.628774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.628709    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerStarted","Data":"6a09516ac35c29f1b3a05552347b0b439a0de4979127aafd27b2e10aabced2b7"}
Jul 02 13:14:51.639183 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.639060    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmjck" event={"ID":"3590789f-cff0-43ff-8158-e0f5ab08ea49","Type":"ContainerStarted","Data":"e5f6243f90724a73a424dbcf5d10a38de0da088014e2aff02360c6235d0f7408"}
Jul 02 13:14:51.645681 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.645507    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal" event={"ID":"1294756ed9becdbbc18223f82bfcc14a","Type":"ContainerStarted","Data":"60fa438d6e442011352e6f37593c6dd9c968fe92c4203653560864f5f9fd2431"}
Jul 02 13:14:51.653814 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.652801    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"c9665c8ced5c59cb8ba888d24dcb89bba2329dfc435d8262883c0239361e7b26"}
Jul 02 13:14:51.656564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.656384    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8qvd" event={"ID":"37e92131-f21d-4309-9f51-9215d2221123","Type":"ContainerStarted","Data":"3f864b512710e1d2bf49b410c038320a6681c74ae91fc3eb81e40a9628e1c00d"}
Jul 02 13:14:51.661299 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:51.661260    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal" event={"ID":"d0695f0d6e8402fe098c22a36367da2e","Type":"ContainerStarted","Data":"8b09052483fc9b0ae4925286cbf4ec6f75d87fc160915a96dd48ec973b4550d7"}
Jul 02 13:14:52.149502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.149465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:52.149697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.149562    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:52.149697 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.149616    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:52.149697 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.149693    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.1496712 +0000 UTC m=+6.005064081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:52.150236 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.150216    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:52.150332 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.150288    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.150268649 +0000 UTC m=+6.005661519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:52.250901 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.250852    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:52.251083 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.251012    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:52.251083 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.251037    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:52.251083 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.251049    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:52.251255 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.251112    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.251094549 +0000 UTC m=+6.106487431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:52.560661 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.560627    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:52.560834 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.560753    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:14:52.561079 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.561056    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:52.561194 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.561081    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:52.561194 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.561174    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:14:52.561313 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:52.561277    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:14:52.587424 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.587274    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:50 +0000 UTC" deadline="2028-01-14 07:48:11.614044383 +0000 UTC"
Jul 02 13:14:52.587424 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:52.587343    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13458h33m19.026707136s"
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.168474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.168563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.168695    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.168759    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.168739874 +0000 UTC m=+10.024132759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.169203    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:54.169306 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.169271    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.169244875 +0000 UTC m=+10.024637954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:54.269979 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.269474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:54.269979 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.269685    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:54.269979 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.269706    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:54.269979 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.269719    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:54.269979 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.269781    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.2697612 +0000 UTC m=+10.125154068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:54.560932 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.560841    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:54.561149 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.561064    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:14:54.561505 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.561462    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:54.561606 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.561558    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:14:54.561663 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:54.561612    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:54.561721 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:54.561703    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:14:56.561026 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:56.560942    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:56.561026 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:56.560966    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:56.561521 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:56.561086    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:14:56.561521 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:56.561097    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:56.561521 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:56.561231    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:14:56.561521 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:56.561350    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:14:58.200327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.200292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:58.200796 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.200353    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:58.200796 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.200510    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.200796 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.200572    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.200553751 +0000 UTC m=+18.055946619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.201057 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.200864    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.201057 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.200951    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.200933566 +0000 UTC m=+18.056326442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.301359 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.300768    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:58.301359 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.300939    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:58.301359 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.300959    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:58.301359 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.300971    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.301359 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.301032    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.301013846 +0000 UTC m=+18.156406714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.561061 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.561025    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:14:58.561248 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.561150    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:14:58.561534 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.561515    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:14:58.561651 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.561632    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:14:58.561692 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:14:58.561679    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:14:58.561764 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:14:58.561746    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:00.561097    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:00.561233    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:00.561295    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:00.561385    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:00.561447    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:00.561643 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:00.561505    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:02.560500 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:02.560475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:02.560500 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:02.560495    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:02.560905 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:02.560477    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:02.560905 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:02.560569    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:02.560905 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:02.560656    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:02.560905 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:02.560714    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:04.560444 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:04.560408    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:04.560444 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:04.560444    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:04.560966 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:04.560410    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:04.560966 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:04.560520    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:04.560966 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:04.560639    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:04.560966 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:04.560729    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:06.262926 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.262888    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:06.263330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.262948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:06.263330 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.263032    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:06.263330 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.263081    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:15:22.263067413 +0000 UTC m=+34.118460280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:06.263330 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.263032    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:06.263330 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.263153    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:22.263139456 +0000 UTC m=+34.118532333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:06.363766 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.363724    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:06.363942 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.363899    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:06.363942 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.363921    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:06.363942 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.363932    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:06.364089 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.363987    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:22.363974307 +0000 UTC m=+34.219367169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:06.560796 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.560718    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:06.560966 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.560859    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:06.560966 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:06.560886    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:06.560966 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.560863    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:06.561123 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.560986    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:06.561123 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:06.561071    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:08.561332 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:08.561063    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:08.561846 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:08.561105    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:08.561846 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:08.561446    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:08.561846 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:08.561515    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:08.561846 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:08.561063    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:08.561846 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:08.561615    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:10.561200 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:10.561172    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:10.561644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:10.561218    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:10.561644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:10.561218    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:10.561644 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:10.561300    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:10.561644 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:10.561397    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:10.561644 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:10.561452    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:12.561006 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.560945    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:12.561006 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.560965    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:12.561690 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.560956    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:12.561690 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:12.561029    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:12.561690 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:12.561100    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:12.561690 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:12.561174    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:12.710752 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.710715    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8wzf" event={"ID":"a04c8f06-90e1-4c77-834b-e9d44cf8bfac","Type":"ContainerStarted","Data":"0a9ad399e2824764cb7e12c7ecdceb27ea5dcdb8bf2ca28ec8cb85313ab56a6f"}
Jul 02 13:15:12.712225 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.712200    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-546hs" event={"ID":"1cb9273a-103d-4f41-b823-1a2c4f6029d5","Type":"ContainerStarted","Data":"28ea1a4adb6f83caf566df1706411feaaa2efa7b522e45367f5c20ef786cf391"}
Jul 02 13:15:12.713619 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.713597    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7" event={"ID":"497e17e1-0fd9-467d-bf51-b261ffe73571","Type":"ContainerStarted","Data":"a425159c27f3179315c578ef8d09ab34398fc6e5047e9f3963634b4cabb13c87"}
Jul 02 13:15:12.714825 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.714806    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvlpw" event={"ID":"354e948f-dc63-416a-8642-8598553c6185","Type":"ContainerStarted","Data":"8b3d26644f299f471baceaa890a968815c2e5f042845998f055fb5c548479feb"}
Jul 02 13:15:12.716216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.716197    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlnbl" event={"ID":"5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e","Type":"ContainerStarted","Data":"3f6f26d69adc50dea226775f7826caa48e3f40e577a942d11a844c82c12c1a10"}
Jul 02 13:15:12.717598 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.717579    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="c8d11d1e301cc371861e3227bc8fdc8e1547fae5b1dbee79061e11704dc5d5be" exitCode=0
Jul 02 13:15:12.717671 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.717639    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"c8d11d1e301cc371861e3227bc8fdc8e1547fae5b1dbee79061e11704dc5d5be"}
Jul 02 13:15:12.719204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.719181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmjck" event={"ID":"3590789f-cff0-43ff-8158-e0f5ab08ea49","Type":"ContainerStarted","Data":"7e325ddef90282af8e5e64ac213215c3d0845436652eccf34da426dd49c1e938"}
Jul 02 13:15:12.720812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.720791    2552 generic.go:356] "Generic (PLEG): container finished" podID="1294756ed9becdbbc18223f82bfcc14a" containerID="68dc350b0a034994e0dede14757aec768212b098638f12eef4348b63f0b40f41" exitCode=0
Jul 02 13:15:12.720903 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.720868    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal" event={"ID":"1294756ed9becdbbc18223f82bfcc14a","Type":"ContainerDied","Data":"68dc350b0a034994e0dede14757aec768212b098638f12eef4348b63f0b40f41"}
Jul 02 13:15:12.723953 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.723934    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"e183c19959d5e5e121023f6f65b57c7d51ba8fadb4a17c1a95e93e324030e9c5"}
Jul 02 13:15:12.724095 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.724004    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"cf87ad6a01d310b0554423f7181f0c1312653a62c74fa22203c742a8ab2fd557"}
Jul 02 13:15:12.724095 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.724033    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"3e75fa274b800e632e6960817718b2d46788ed5c95b9a6b59b5bc0590021e3d9"}
Jul 02 13:15:12.724095 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.724078    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"dce68fc13a7276d59a03d0407edff5c768411ed0064c8814e304c2cdf091cd74"}
Jul 02 13:15:12.724229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.724115    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"d93ea015e1e3348d35815e8a0241c3ce1143f4737ef8c612b9a2b9d2a0e356f5"}
Jul 02 13:15:12.724229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.724156    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"b2fd95bb650d8592b35a179a880b2d15063c7b582b5100470db102ebb227ef5e"}
Jul 02 13:15:12.725251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.725234    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal" event={"ID":"d0695f0d6e8402fe098c22a36367da2e","Type":"ContainerStarted","Data":"e61423f2d066e3afe1c29a80cdd744d7d0d58d2779fde81b8db4e6879f6c637e"}
Jul 02 13:15:12.783087 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.782901    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r8wzf" podStartSLOduration=6.861277527 podStartE2EDuration="22.782885034s" totalImagesPullingTime="15.921607507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.782522811 +0000 UTC m=+24.637915720" watchObservedRunningTime="2026-07-02 13:15:12.782885034 +0000 UTC m=+24.638277911"
Jul 02 13:15:12.864107 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.864057    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tvlpw" podStartSLOduration=2.102145131 podStartE2EDuration="22.864042719s" totalImagesPullingTime="20.761897588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.863526431 +0000 UTC m=+24.718919317" watchObservedRunningTime="2026-07-02 13:15:12.864042719 +0000 UTC m=+24.719435605"
Jul 02 13:15:12.937014 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.936969    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gmjck" podStartSLOduration=1.97623391 podStartE2EDuration="22.936955823s" totalImagesPullingTime="20.960721913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.900737206 +0000 UTC m=+24.756130080" watchObservedRunningTime="2026-07-02 13:15:12.936955823 +0000 UTC m=+24.792348763"
Jul 02 13:15:12.937162 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.937143    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rlnbl" podStartSLOduration=1.937369321 podStartE2EDuration="22.937138182s" totalImagesPullingTime="20.999768861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.937084198 +0000 UTC m=+24.792477094" watchObservedRunningTime="2026-07-02 13:15:12.937138182 +0000 UTC m=+24.792531065"
Jul 02 13:15:12.975670 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:12.975633    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-546hs" podStartSLOduration=2.146238988 podStartE2EDuration="22.975622155s" totalImagesPullingTime="20.829383167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.975320597 +0000 UTC m=+24.830713505" watchObservedRunningTime="2026-07-02 13:15:12.975622155 +0000 UTC m=+24.831015038"
Jul 02 13:15:13.043145 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:13.043108    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-149.ec2.internal" podStartSLOduration=23.043096016 podStartE2EDuration="23.043096016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:13.04296086 +0000 UTC m=+24.898353745" watchObservedRunningTime="2026-07-02 13:15:13.043096016 +0000 UTC m=+24.898488899"
Jul 02 13:15:13.730582 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:13.730547    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal" event={"ID":"1294756ed9becdbbc18223f82bfcc14a","Type":"ContainerStarted","Data":"862cdd8ac00660cc6372c0f40006750c04876f0c0a9e8023377914902f7b63b4"}
Jul 02 13:15:13.732521 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:13.732456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8qvd" event={"ID":"37e92131-f21d-4309-9f51-9215d2221123","Type":"ContainerStarted","Data":"2ef9ed56592737a823d696b1a74cec335cc3b7345379759af15d971c42d41125"}
Jul 02 13:15:13.758015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:13.757968    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-149.ec2.internal" podStartSLOduration=23.757951457 podStartE2EDuration="23.757951457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:13.757400833 +0000 UTC m=+25.612793718" watchObservedRunningTime="2026-07-02 13:15:13.757951457 +0000 UTC m=+25.613344342"
Jul 02 13:15:14.075672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.075630    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:15:14.309146 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.309066    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:15:14.309618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.309595    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:15:14.335671 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.335620    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g8qvd" podStartSLOduration=3.536386147 podStartE2EDuration="24.335601283s" totalImagesPullingTime="20.799215136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:13.786901471 +0000 UTC m=+25.642294376" watchObservedRunningTime="2026-07-02 13:15:14.335601283 +0000 UTC m=+26.190994168"
Jul 02 13:15:14.544800 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.544711    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:15:14.075651555Z","UUID":"76f061e5-ab3e-47fb-a0e2-0be1857b085b","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:15:14.546494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.546470    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:15:14.546614 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.546523    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:15:14.561078 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.561021    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:14.561078 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.561044    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:14.561078 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.561044    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:14.561255 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:14.561113    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:14.561255 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:14.561226    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:14.561332 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:14.561287    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:14.739654 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.739560    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7" event={"ID":"497e17e1-0fd9-467d-bf51-b261ffe73571","Type":"ContainerStarted","Data":"78be81757105a7cc19a0b66372724b20aa396542d4c37e1d0c4facbd4be42d56"}
Jul 02 13:15:14.748246 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.747893    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"b0cfca7ae021d5f1f72765276ea8067a38d9744cf09874142303378f7cfe45e9"}
Jul 02 13:15:14.748246 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.748033    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:15:14.748499 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:14.748352    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tvlpw"
Jul 02 13:15:15.752068 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:15.752026    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7" event={"ID":"497e17e1-0fd9-467d-bf51-b261ffe73571","Type":"ContainerStarted","Data":"298a8c7483f77659dd9650166399813c0c09a1f015e23f50e491c09fdecca8a8"}
Jul 02 13:15:15.784183 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:15.784138    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5dxc7" podStartSLOduration=2.255835792 podStartE2EDuration="25.784117318s" totalImagesPullingTime="23.528281526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:15.783626313 +0000 UTC m=+27.639019200" watchObservedRunningTime="2026-07-02 13:15:15.784117318 +0000 UTC m=+27.639510201"
Jul 02 13:15:16.561173 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:16.561142    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:16.561173 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:16.561171    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:16.561372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:16.561230    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:16.561372 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:16.561318    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:16.561450 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:16.561395    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:16.561450 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:16.561436    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:17.757721 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.757542    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="4498bdf00988e7957a8168a41e3b9a433653426ac46b5a3ec2571e35a19162c7" exitCode=0
Jul 02 13:15:17.758473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.757619    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"4498bdf00988e7957a8168a41e3b9a433653426ac46b5a3ec2571e35a19162c7"}
Jul 02 13:15:17.761517 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.761494    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" event={"ID":"e636dd3e-e640-47e6-8d2f-b16930340638","Type":"ContainerStarted","Data":"a8dfd431c1c484044707058ffecaf02551ed45b7fea950a10a5192aed3185011"}
Jul 02 13:15:17.761741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.761719    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:17.761847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.761774    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:17.776545 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:17.776521    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:18.560471 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.560441    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:18.560471 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.560469    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:18.560736 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.560473    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:18.560736 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:18.560561    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:18.560736 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:18.560615    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:18.560736 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:18.560720    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:18.763921 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.763894    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:18.778053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.778031    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:18.812442 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:18.812349    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg" podStartSLOduration=7.85315952 podStartE2EDuration="28.81233273s" totalImagesPullingTime="20.95917321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:17.824006091 +0000 UTC m=+29.679398975" watchObservedRunningTime="2026-07-02 13:15:18.81233273 +0000 UTC m=+30.667725593"
Jul 02 13:15:19.767603 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:19.767570    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="373497cefbe813ec9868ef077ff5fa2b078076886d24d4d4abc42e4ccd24e180" exitCode=0
Jul 02 13:15:19.768233 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:19.767656    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"373497cefbe813ec9868ef077ff5fa2b078076886d24d4d4abc42e4ccd24e180"}
Jul 02 13:15:20.560586 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:20.560557    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:20.560755 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:20.560592    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:20.560755 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:20.560561    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:20.560755 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:20.560662    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:20.560755 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:20.560737    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:20.560930 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:20.560796    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:21.774311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:21.774281    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerStarted","Data":"0b8cbb0cfef0791f29a780e859f766c5fa46b416b0a52f5eb3b955346b56a3ab"}
Jul 02 13:15:22.284598 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.284568    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:22.284749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.284606    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:22.284749 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.284701    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:22.284749 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.284703    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:22.284749 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.284747    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs podName:c86e3245-2705-4a72-b608-61bebd9eb17e nodeName:}" failed. No retries permitted until 2026-07-02 13:15:54.284734793 +0000 UTC m=+66.140127655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs") pod "network-metrics-daemon-q9mnj" (UID: "c86e3245-2705-4a72-b608-61bebd9eb17e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:22.284939 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.284759    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret podName:3f6ceabc-fa61-470e-8df7-db311b0433f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:54.284753665 +0000 UTC m=+66.140146526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret") pod "global-pull-secret-syncer-stzdv" (UID: "3f6ceabc-fa61-470e-8df7-db311b0433f6") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:22.385700 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.385674    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:22.385826 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.385792    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:22.385826 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.385807    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:22.385826 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.385815    2552 projected.go:196] Error preparing data for projected volume kube-api-access-ghff8 for pod openshift-network-diagnostics/network-check-target-w5957: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:22.385958 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.385855    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8 podName:6354334f-1524-4060-948b-c568d9382894 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:54.385843983 +0000 UTC m=+66.241236848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghff8" (UniqueName: "kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8") pod "network-check-target-w5957" (UID: "6354334f-1524-4060-948b-c568d9382894") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:22.561009 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.560933    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:22.561009 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.560964    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:22.561209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.561028    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:22.561209 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.561042    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:22.561209 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.561099    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:22.561209 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:22.561160    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:22.778511 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.778476    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="0b8cbb0cfef0791f29a780e859f766c5fa46b416b0a52f5eb3b955346b56a3ab" exitCode=0
Jul 02 13:15:22.778982 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:22.778533    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"0b8cbb0cfef0791f29a780e859f766c5fa46b416b0a52f5eb3b955346b56a3ab"}
Jul 02 13:15:24.560860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:24.560830    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:24.561285 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:24.560830    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:24.561285 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:24.560966    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:24.561285 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:24.560830    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:24.561285 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:24.561042    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:24.561285 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:24.561091    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:26.560811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:26.560603    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:26.561253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:26.560603    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:26.561253 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:26.560903    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:26.561253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:26.560604    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:26.561253 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:26.560987    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:26.561253 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:26.561066    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:26.790109 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:26.790068    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerStarted","Data":"3731bdd6999166ade14edb17d38e7f7bb58a4ee94d497798e80755c377310e7f"}
Jul 02 13:15:27.795006 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:27.794976    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="3731bdd6999166ade14edb17d38e7f7bb58a4ee94d497798e80755c377310e7f" exitCode=0
Jul 02 13:15:27.795349 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:27.795041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"3731bdd6999166ade14edb17d38e7f7bb58a4ee94d497798e80755c377310e7f"}
Jul 02 13:15:28.560551 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:28.560520    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:28.560741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:28.560520    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:28.560741 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:28.560611    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:28.560741 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:28.560726    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:28.560944 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:28.560520    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:28.560944 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:28.560844    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:28.799792 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:28.799765    2552 generic.go:356] "Generic (PLEG): container finished" podID="a62feb9f-c762-41b5-804b-7abe8c0facac" containerID="6ec496eac3c5319aa1d1773bce54a99c0d577b93165597767ae2f7d9de5f779e" exitCode=0
Jul 02 13:15:28.800164 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:28.799822    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerDied","Data":"6ec496eac3c5319aa1d1773bce54a99c0d577b93165597767ae2f7d9de5f779e"}
Jul 02 13:15:29.804973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:29.804935    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s89wm" event={"ID":"a62feb9f-c762-41b5-804b-7abe8c0facac","Type":"ContainerStarted","Data":"330305369b4c5caf5a12bcbf78618db2293751f64459010846fecdcac9fc03ab"}
Jul 02 13:15:29.837830 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:29.837788    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s89wm" podStartSLOduration=7.470327388 podStartE2EDuration="39.837772636s" totalImagesPullingTime="32.367445248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:29.837318684 +0000 UTC m=+41.692711568" watchObservedRunningTime="2026-07-02 13:15:29.837772636 +0000 UTC m=+41.693165527"
Jul 02 13:15:30.560909 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:30.560868    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:30.561060 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:30.560884    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:30.561060 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:30.560986    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:30.561060 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:30.560869    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:30.561189 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:30.561056    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:30.561189 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:30.561115    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:32.561067 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:32.561035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:32.561411 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:32.561035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:32.561411 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:32.561134    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:32.561411 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:32.561035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:32.561411 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:32.561197    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:32.561411 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:32.561270    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:34.560930 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:34.560898    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:34.561379 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:34.560898    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:34.561379 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:34.561012    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:34.561379 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:34.560901    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:34.561379 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:34.561073    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:34.561379 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:34.561142    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:36.561033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.561005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:36.561369 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.561004    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:36.561369 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.561120    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:36.561369 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.561208    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:36.561369 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.561006    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:36.561369 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.561307    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:36.745842 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.745810    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q9mnj"]
Jul 02 13:15:36.749606 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.749582    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5957"]
Jul 02 13:15:36.783758 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.783699    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stzdv"]
Jul 02 13:15:36.818283 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.818248    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:36.818283 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.818261    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:36.818463 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:36.818261    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:36.818463 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.818373    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:36.818544 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.818497    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:36.818580 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:36.818568    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:38.561336 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:38.561170    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:38.561703 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:38.561178    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:38.561703 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:38.561404    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:38.561703 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:38.561178    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:38.561703 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:38.561479    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:38.561703 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:38.561544    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:40.560286 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:40.560243    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:40.560706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:40.560252    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:40.560706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:40.560363    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:40.560706 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:40.560357    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9mnj" podUID="c86e3245-2705-4a72-b608-61bebd9eb17e"
Jul 02 13:15:40.560706 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:40.560442    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stzdv" podUID="3f6ceabc-fa61-470e-8df7-db311b0433f6"
Jul 02 13:15:40.560706 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:40.560533    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5957" podUID="6354334f-1524-4060-948b-c568d9382894"
Jul 02 13:15:41.369664 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.369641    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-149.ec2.internal" event="NodeReady"
Jul 02 13:15:41.369764 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.369760    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:15:41.419654 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.419629    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-92cmb"]
Jul 02 13:15:41.430866 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.430847    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-26spv"]
Jul 02 13:15:41.430981 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.430964    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.433498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.433476    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:15:41.433498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.433492    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xss9w\""
Jul 02 13:15:41.433747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.433734    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:15:41.441095 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.441077    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-92cmb"]
Jul 02 13:15:41.441181 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.441112    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-26spv"]
Jul 02 13:15:41.441246 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.441220    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.444486 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.444464    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:15:41.444486 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.444465    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwhxh\""
Jul 02 13:15:41.444975 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.444956    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:15:41.445159 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.444960    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:15:41.445238 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.445057    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:15:41.524232 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.524208    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w57dk"]
Jul 02 13:15:41.530749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cdca863-8d86-4583-8838-4607100cc261-crio-socket\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.530811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-metrics-tls\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.530811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-tmp-dir\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.530811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530800    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-config-volume\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.530936 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtrg\" (UniqueName: \"kubernetes.io/projected/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-kube-api-access-twtrg\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.530970 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530944    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdca863-8d86-4583-8838-4607100cc261-data-volume\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.531003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.530974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cdca863-8d86-4583-8838-4607100cc261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.531032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.531000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cdca863-8d86-4583-8838-4607100cc261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.531063 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.531051    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb88n\" (UniqueName: \"kubernetes.io/projected/3cdca863-8d86-4583-8838-4607100cc261-kube-api-access-zb88n\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.545054 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.545033    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w57dk"]
Jul 02 13:15:41.545149 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.545083    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.548516 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.548486    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:15:41.548516 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.548499    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vqqgk\""
Jul 02 13:15:41.548669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.548565    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:15:41.548709 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.548686    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:15:41.631812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631756    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cdca863-8d86-4583-8838-4607100cc261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.631812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb88n\" (UniqueName: \"kubernetes.io/projected/3cdca863-8d86-4583-8838-4607100cc261-kube-api-access-zb88n\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cdca863-8d86-4583-8838-4607100cc261-crio-socket\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-metrics-tls\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631911    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cdca863-8d86-4583-8838-4607100cc261-crio-socket\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631922    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720c4f68-b38c-4924-9d7f-f027bf16cdd8-cert\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631944    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-tmp-dir\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631972    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-config-volume\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.631992    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtrg\" (UniqueName: \"kubernetes.io/projected/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-kube-api-access-twtrg\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdca863-8d86-4583-8838-4607100cc261-data-volume\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cdca863-8d86-4583-8838-4607100cc261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632191    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdkt\" (UniqueName: \"kubernetes.io/projected/720c4f68-b38c-4924-9d7f-f027bf16cdd8-kube-api-access-vsdkt\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.632759 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632360    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-tmp-dir\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.632759 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632370    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cdca863-8d86-4583-8838-4607100cc261-data-volume\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632759 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632553    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cdca863-8d86-4583-8838-4607100cc261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.632759 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.632669    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-config-volume\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.635734 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.635717    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cdca863-8d86-4583-8838-4607100cc261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.635734 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.635725    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-metrics-tls\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.642976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.642953    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb88n\" (UniqueName: \"kubernetes.io/projected/3cdca863-8d86-4583-8838-4607100cc261-kube-api-access-zb88n\") pod \"insights-runtime-extractor-26spv\" (UID: \"3cdca863-8d86-4583-8838-4607100cc261\") " pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.643331 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.643316    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtrg\" (UniqueName: \"kubernetes.io/projected/ee0f85d9-e141-4201-b5fc-c1bc593f6fc7-kube-api-access-twtrg\") pod \"dns-default-92cmb\" (UID: \"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7\") " pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.732467 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.732446    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdkt\" (UniqueName: \"kubernetes.io/projected/720c4f68-b38c-4924-9d7f-f027bf16cdd8-kube-api-access-vsdkt\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.732543 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.732487    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720c4f68-b38c-4924-9d7f-f027bf16cdd8-cert\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.734349 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.734334    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720c4f68-b38c-4924-9d7f-f027bf16cdd8-cert\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.740717 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.740699    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:41.741298 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.741279    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdkt\" (UniqueName: \"kubernetes.io/projected/720c4f68-b38c-4924-9d7f-f027bf16cdd8-kube-api-access-vsdkt\") pod \"ingress-canary-w57dk\" (UID: \"720c4f68-b38c-4924-9d7f-f027bf16cdd8\") " pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.750058 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.750043    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-26spv"
Jul 02 13:15:41.855541 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.855299    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w57dk"
Jul 02 13:15:41.963706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.963665    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-92cmb"]
Jul 02 13:15:41.971722 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:41.971600    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0f85d9_e141_4201_b5fc_c1bc593f6fc7.slice/crio-29b9255b2c95a1d4c6a3e2a7ef96ba4815cc3f8f1481358c8a206d104c2f309d WatchSource:0}: Error finding container 29b9255b2c95a1d4c6a3e2a7ef96ba4815cc3f8f1481358c8a206d104c2f309d: Status 404 returned error can't find the container with id 29b9255b2c95a1d4c6a3e2a7ef96ba4815cc3f8f1481358c8a206d104c2f309d
Jul 02 13:15:41.974530 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:41.974430    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-26spv"]
Jul 02 13:15:41.978308 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:41.978288    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdca863_8d86_4583_8838_4607100cc261.slice/crio-12e620d5b5953fbf6c4079e3f28eb05555ef377485780c03aa49af582ae642f4 WatchSource:0}: Error finding container 12e620d5b5953fbf6c4079e3f28eb05555ef377485780c03aa49af582ae642f4: Status 404 returned error can't find the container with id 12e620d5b5953fbf6c4079e3f28eb05555ef377485780c03aa49af582ae642f4
Jul 02 13:15:42.067656 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.067630    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w57dk"]
Jul 02 13:15:42.070042 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:42.070016    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720c4f68_b38c_4924_9d7f_f027bf16cdd8.slice/crio-c59ce22d6e4197e82435c805099b8b62a5e8976c9ca10da012af6f5ecea42f68 WatchSource:0}: Error finding container c59ce22d6e4197e82435c805099b8b62a5e8976c9ca10da012af6f5ecea42f68: Status 404 returned error can't find the container with id c59ce22d6e4197e82435c805099b8b62a5e8976c9ca10da012af6f5ecea42f68
Jul 02 13:15:42.533261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.533102    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:15:42.560286 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.560266    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:15:42.560384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.560346    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.560477 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.560442    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:42.560684 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.560441    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:42.560811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.560445    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:42.563063 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563040    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lw488\""
Jul 02 13:15:42.563063 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563055    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:15:42.563166 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563039    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:15:42.563749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563734    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:15:42.563846 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563831    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:15:42.563920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563858    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:15:42.563920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563836    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:42.564024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563836    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:42.564024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.563943    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-54grc\""
Jul 02 13:15:42.564024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.564020    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:42.564328 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.564310    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zp2gh\""
Jul 02 13:15:42.564394 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.564347    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:15:42.564445 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.564403    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:42.572341 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.572325    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:15:42.637341 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.637341 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637339    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.638021 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.638021 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpg7\" (UniqueName: \"kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.638021 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637440    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.638021 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.637454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738717    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpg7\" (UniqueName: \"kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738781    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738852    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738897    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.738927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.738922    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.739720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.739660    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.739720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.739682    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.739720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.739662    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.742098 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.742076    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.742237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.742213    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.747731 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.747707    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpg7\" (UniqueName: \"kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7\") pod \"console-5b9bbcdfbc-4gns6\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") " pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:42.832615 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.832517    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w57dk" event={"ID":"720c4f68-b38c-4924-9d7f-f027bf16cdd8","Type":"ContainerStarted","Data":"c59ce22d6e4197e82435c805099b8b62a5e8976c9ca10da012af6f5ecea42f68"}
Jul 02 13:15:42.833887 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.833846    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-26spv" event={"ID":"3cdca863-8d86-4583-8838-4607100cc261","Type":"ContainerStarted","Data":"12e620d5b5953fbf6c4079e3f28eb05555ef377485780c03aa49af582ae642f4"}
Jul 02 13:15:42.835007 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.834983    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-92cmb" event={"ID":"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7","Type":"ContainerStarted","Data":"29b9255b2c95a1d4c6a3e2a7ef96ba4815cc3f8f1481358c8a206d104c2f309d"}
Jul 02 13:15:42.870490 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:42.870464    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:43.043753 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:43.043715    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:15:43.047825 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:43.047799    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c22607d_25f7_4a17_9d52_00eab5364312.slice/crio-b844b8728d8b6b1fbf3c5f1b81031dafa25c0cffcd58baa414bb9626666ca69f WatchSource:0}: Error finding container b844b8728d8b6b1fbf3c5f1b81031dafa25c0cffcd58baa414bb9626666ca69f: Status 404 returned error can't find the container with id b844b8728d8b6b1fbf3c5f1b81031dafa25c0cffcd58baa414bb9626666ca69f
Jul 02 13:15:43.840161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:43.840104    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9bbcdfbc-4gns6" event={"ID":"1c22607d-25f7-4a17-9d52-00eab5364312","Type":"ContainerStarted","Data":"b844b8728d8b6b1fbf3c5f1b81031dafa25c0cffcd58baa414bb9626666ca69f"}
Jul 02 13:15:45.352886 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.352849    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"]
Jul 02 13:15:45.368769 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.368689    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-ctlsz"]
Jul 02 13:15:45.369685 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.369042    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.372135 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.371916    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:15:45.372135 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.371985    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:15:45.372135 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.372067    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:15:45.372344 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.372223    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n75bq\""
Jul 02 13:15:45.372934 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.372851    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:15:45.373033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.372932    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 13:15:45.390578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.390554    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"]
Jul 02 13:15:45.390671 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.390635    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.393267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.393251    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 13:15:45.393387 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.393363    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 13:15:45.393452 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.393436    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7mpkp\""
Jul 02 13:15:45.393912 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.393764    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:15:45.406440 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.405401    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-ctlsz"]
Jul 02 13:15:45.406440 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.405456    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mmmhc"]
Jul 02 13:15:45.427989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.427832    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.431279 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.431259    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:15:45.431367 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.431281    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:15:45.431574 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.431554    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9lg6\""
Jul 02 13:15:45.431774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.431756    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:15:45.463996 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.463963    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-root\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.463996 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.463993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7vj\" (UniqueName: \"kubernetes.io/projected/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-kube-api-access-qm7vj\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-metrics-client-ca\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-metrics-client-ca\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464126    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.464186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464216    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-sys\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464240    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464273    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgld\" (UniqueName: \"kubernetes.io/projected/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-kube-api-access-2xgld\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.464361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464346    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464369    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxz9\" (UniqueName: \"kubernetes.io/projected/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-api-access-xjxz9\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464391    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a472ad0-3f79-4836-b998-eaa7ddf9f939-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464423    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464456    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-wtmp\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464482    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-tls\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464503    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.464785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464581    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-textfile\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.464785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.464627    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.565631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565594    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a472ad0-3f79-4836-b998-eaa7ddf9f939-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.565799 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.565799 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565694    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-wtmp\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.565799 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-tls\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.565799 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565745    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566036 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:45.565851    2552 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 02 13:15:45.566036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565912    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-wtmp\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566036 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:45.565943    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls podName:e7a15b67-891f-40f8-a64b-30bb3a2ce04e nodeName:}" failed. No retries permitted until 2026-07-02 13:15:46.065922037 +0000 UTC m=+57.921314907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls") pod "openshift-state-metrics-57f99c8b49-jc98n" (UID: "e7a15b67-891f-40f8-a64b-30bb3a2ce04e") : secret "openshift-state-metrics-tls" not found
Jul 02 13:15:45.566036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.565967    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-textfile\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566005    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566040    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a472ad0-3f79-4836-b998-eaa7ddf9f939-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-root\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566110    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7vj\" (UniqueName: \"kubernetes.io/projected/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-kube-api-access-qm7vj\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-metrics-client-ca\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566172    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-metrics-client-ca\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566198    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566229    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566289 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-sys\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566321    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566372    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgld\" (UniqueName: \"kubernetes.io/projected/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-kube-api-access-2xgld\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.566667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566401    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566426    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxz9\" (UniqueName: \"kubernetes.io/projected/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-api-access-xjxz9\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.566939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566812    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-metrics-client-ca\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566995 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566941    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-root\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.566995 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.566320    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-textfile\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.567092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.567005    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-sys\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.567092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.567053    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.567412 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:45.567301    2552 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 02 13:15:45.567412 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:45.567378    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls podName:0a472ad0-3f79-4836-b998-eaa7ddf9f939 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:46.067362714 +0000 UTC m=+57.922755595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-ctlsz" (UID: "0a472ad0-3f79-4836-b998-eaa7ddf9f939") : secret "kube-state-metrics-tls" not found
Jul 02 13:15:45.567577 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.567526    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-metrics-client-ca\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.567644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.567619    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.567943 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.567853    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.569916 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.569591    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.569916 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.569851    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-node-exporter-tls\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.569916 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.569851    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.570237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.570218    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.578051 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.578025    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxz9\" (UniqueName: \"kubernetes.io/projected/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-api-access-xjxz9\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:45.578151 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.578101    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgld\" (UniqueName: \"kubernetes.io/projected/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-kube-api-access-2xgld\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:45.579518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.579475    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7vj\" (UniqueName: \"kubernetes.io/projected/7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b-kube-api-access-qm7vj\") pod \"node-exporter-mmmhc\" (UID: \"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b\") " pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:45.738726 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:45.738689    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mmmhc"
Jul 02 13:15:46.072568 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.072491    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:46.072733 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.072579    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:46.075198 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.075171    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7a15b67-891f-40f8-a64b-30bb3a2ce04e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-jc98n\" (UID: \"e7a15b67-891f-40f8-a64b-30bb3a2ce04e\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:46.075318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.075271    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a472ad0-3f79-4836-b998-eaa7ddf9f939-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-ctlsz\" (UID: \"0a472ad0-3f79-4836-b998-eaa7ddf9f939\") " pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:46.284007 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.283983    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"
Jul 02 13:15:46.301124 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.301103    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz"
Jul 02 13:15:46.377426 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.377361    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:46.381296 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.381275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.384525 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.384208    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 13:15:46.384525 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.384478    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 13:15:46.384682 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.384544    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 13:15:46.384836 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.384815    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-px427\""
Jul 02 13:15:46.385636 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.385063    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 13:15:46.385636 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.385536    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 13:15:46.388627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.385956    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 13:15:46.388627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.386219    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 13:15:46.388627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.385965    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 13:15:46.388627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.387246    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 13:15:46.398360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.398339    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:46.476155 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476126    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476312 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476312 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476227    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476312 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476265    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476312 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476297    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476320    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476413    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476446    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mq7n\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476464    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.476860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.476544    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577557 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577557 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577560    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577765    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mq7n\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577857 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577820    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.577929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.577912    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578152 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:46.578128    2552 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 02 13:15:46.578279 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:46.578204    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls podName:a2b0d777-c27a-4d59-9a07-5fc0a65c8655 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:47.078182328 +0000 UTC m=+58.933575219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655") : secret "alertmanager-main-tls" not found
Jul 02 13:15:46.578279 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578415 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578310    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578415 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578343    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578415 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578556 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578556 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578556 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578496    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578763 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578661    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.578817 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.578774    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.579275 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.579247    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.580366 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.580339    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.580891 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.580824    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.581189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.581160    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.581604 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.581584    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.582209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.582017    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.582209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.582164    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.582350 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.582248    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.582550 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.582532    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:46.590746 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:46.590716    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mq7n\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:47.083581 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:47.083549    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:47.086044 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:47.086023    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:47.295550 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:47.295472    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:48.311003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.310975    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ccd878888-872p5"]
Jul 02 13:15:48.318493 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.318475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.322599 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.322565    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 13:15:48.322724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.322703    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 13:15:48.322808 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.322790    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 13:15:48.323150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.323127    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tbnhj\""
Jul 02 13:15:48.323252 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.323175    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 13:15:48.323252 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.323175    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-895frpknvcag4\""
Jul 02 13:15:48.323960 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.323945    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 13:15:48.330723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.330704    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ccd878888-872p5"]
Jul 02 13:15:48.397187 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397115    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397187 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397199    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282fcef4-e4e9-4b69-8edf-62c585b3ffca-metrics-client-ca\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397237    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397263    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397278    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8qk\" (UniqueName: \"kubernetes.io/projected/282fcef4-e4e9-4b69-8edf-62c585b3ffca-kube-api-access-8h8qk\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397298    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.397383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.397322    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-grpc-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498626 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498594    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498626 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498629    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282fcef4-e4e9-4b69-8edf-62c585b3ffca-metrics-client-ca\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498666    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498700    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8qk\" (UniqueName: \"kubernetes.io/projected/282fcef4-e4e9-4b69-8edf-62c585b3ffca-kube-api-access-8h8qk\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498756    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498780    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-grpc-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.498850 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.498840    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.499395 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.499370    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282fcef4-e4e9-4b69-8edf-62c585b3ffca-metrics-client-ca\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.501418 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501386    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 13:15:48.501535 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501428    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 13:15:48.501535 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501399    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 13:15:48.501535 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501460    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-895frpknvcag4\""
Jul 02 13:15:48.501535 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501510    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 13:15:48.501724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.501546    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 13:15:48.507761 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.507739    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8qk\" (UniqueName: \"kubernetes.io/projected/282fcef4-e4e9-4b69-8edf-62c585b3ffca-kube-api-access-8h8qk\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.512073 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.512020    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.512073 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.512064    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.512410 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.512367    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.512629 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.512612    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-grpc-tls\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.513103 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.513082    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.513351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.513123    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/282fcef4-e4e9-4b69-8edf-62c585b3ffca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ccd878888-872p5\" (UID: \"282fcef4-e4e9-4b69-8edf-62c585b3ffca\") " pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:48.631555 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.631528    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tbnhj\""
Jul 02 13:15:48.639451 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:48.639433    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:49.180320 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.177442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-ctlsz"]
Jul 02 13:15:49.180320 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.179631    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n"]
Jul 02 13:15:49.184116 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:49.184022    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a472ad0_3f79_4836_b998_eaa7ddf9f939.slice/crio-c8bddd309e9ca79a084beb0fbb514b56f3394d99447b6f4937b9f6fa88c03196 WatchSource:0}: Error finding container c8bddd309e9ca79a084beb0fbb514b56f3394d99447b6f4937b9f6fa88c03196: Status 404 returned error can't find the container with id c8bddd309e9ca79a084beb0fbb514b56f3394d99447b6f4937b9f6fa88c03196
Jul 02 13:15:49.189551 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:49.189431    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a15b67_891f_40f8_a64b_30bb3a2ce04e.slice/crio-9fec6cf2acecb92479c01350ab29fcf6f6a87a3d19cd7a63853d665bdc25a499 WatchSource:0}: Error finding container 9fec6cf2acecb92479c01350ab29fcf6f6a87a3d19cd7a63853d665bdc25a499: Status 404 returned error can't find the container with id 9fec6cf2acecb92479c01350ab29fcf6f6a87a3d19cd7a63853d665bdc25a499
Jul 02 13:15:49.222862 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.222803    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ccd878888-872p5"]
Jul 02 13:15:49.229709 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.229326    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:49.235967 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:49.231587    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282fcef4_e4e9_4b69_8edf_62c585b3ffca.slice/crio-4eb28d8227ce5c1496c433233c28bc15f466caa6c992b8d1783c722c83816287 WatchSource:0}: Error finding container 4eb28d8227ce5c1496c433233c28bc15f466caa6c992b8d1783c722c83816287: Status 404 returned error can't find the container with id 4eb28d8227ce5c1496c433233c28bc15f466caa6c992b8d1783c722c83816287
Jul 02 13:15:49.853953 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.853820    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b57f869fd-ppsvt"]
Jul 02 13:15:49.856906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.856431    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.859731 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.859703    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w57dk" event={"ID":"720c4f68-b38c-4924-9d7f-f027bf16cdd8","Type":"ContainerStarted","Data":"e427fa38d5e27f654dcf05b4776fda1bd53d7dddc4dcc0449357faae32286080"}
Jul 02 13:15:49.861278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.861253    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gl5pg\""
Jul 02 13:15:49.862515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.862479    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 13:15:49.862624 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.862576    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5uv03lckjrtpr\""
Jul 02 13:15:49.862699 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.862625    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 13:15:49.862750 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.862725    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 13:15:49.862798 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.862786    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 13:15:49.863297 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.863268    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-92cmb" event={"ID":"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7","Type":"ContainerStarted","Data":"e10bce932d2f08b48460041b073adeb8ff6b9102df1f23181875ca79d630a6e0"}
Jul 02 13:15:49.863403 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.863324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-92cmb" event={"ID":"ee0f85d9-e141-4201-b5fc-c1bc593f6fc7","Type":"ContainerStarted","Data":"006369d3f0cbb1cadc848323919f4f83b8ce2d5da452c582631366cc6b0a3dea"}
Jul 02 13:15:49.863785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.863735    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:49.868507 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.868474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9bbcdfbc-4gns6" event={"ID":"1c22607d-25f7-4a17-9d52-00eab5364312","Type":"ContainerStarted","Data":"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"}
Jul 02 13:15:49.870465 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.870405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz" event={"ID":"0a472ad0-3f79-4836-b998-eaa7ddf9f939","Type":"ContainerStarted","Data":"c8bddd309e9ca79a084beb0fbb514b56f3394d99447b6f4937b9f6fa88c03196"}
Jul 02 13:15:49.872742 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.872722    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n" event={"ID":"e7a15b67-891f-40f8-a64b-30bb3a2ce04e","Type":"ContainerStarted","Data":"3eb4c5c34da163c8947f59e8116e5a02d75242077ed927eb0b173fb6ce1cbcc1"}
Jul 02 13:15:49.872819 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.872769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n" event={"ID":"e7a15b67-891f-40f8-a64b-30bb3a2ce04e","Type":"ContainerStarted","Data":"e755e8bb3b2e4473763173a7b9102dc4d54414ad463338f28c7acc1345f04303"}
Jul 02 13:15:49.872819 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.872788    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n" event={"ID":"e7a15b67-891f-40f8-a64b-30bb3a2ce04e","Type":"ContainerStarted","Data":"9fec6cf2acecb92479c01350ab29fcf6f6a87a3d19cd7a63853d665bdc25a499"}
Jul 02 13:15:49.874699 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.874676    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-26spv" event={"ID":"3cdca863-8d86-4583-8838-4607100cc261","Type":"ContainerStarted","Data":"d6556fb91af37899f0c3314789f425836e87b4c87f388104cf83550b33334da0"}
Jul 02 13:15:49.876367 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.876334    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"4eb28d8227ce5c1496c433233c28bc15f466caa6c992b8d1783c722c83816287"}
Jul 02 13:15:49.877631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.877606    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"0505fe88c72878f3308987a110ad8edaeb5a27779ed411f4303ce259dc4da182"}
Jul 02 13:15:49.879037 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.879014    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmmhc" event={"ID":"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b","Type":"ContainerStarted","Data":"67b4a72349e41cbb42dd43f970dd7957f9a8096c89c68a019981efc1a0576d98"}
Jul 02 13:15:49.893886 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.893847    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b57f869fd-ppsvt"]
Jul 02 13:15:49.911494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911272    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-metrics-server-audit-profiles\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911307    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdrw\" (UniqueName: \"kubernetes.io/projected/894ee300-2398-4436-977c-82973679ed08-kube-api-access-qsdrw\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911360    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-client-certs\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911375    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/894ee300-2398-4436-977c-82973679ed08-audit-log\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911753 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911636    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911753 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911682    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-tls\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.911753 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.911726    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-client-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:49.934817 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.934627    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-92cmb" podStartSLOduration=1.938688547 podStartE2EDuration="8.934607339s" totalImagesPullingTime="6.995918792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:49.934546811 +0000 UTC m=+61.789939696" watchObservedRunningTime="2026-07-02 13:15:49.934607339 +0000 UTC m=+61.790000224"
Jul 02 13:15:49.971445 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:49.971399    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w57dk" podStartSLOduration=2.406530523 podStartE2EDuration="8.971384424s" totalImagesPullingTime="6.564853901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:49.970765914 +0000 UTC m=+61.826158798" watchObservedRunningTime="2026-07-02 13:15:49.971384424 +0000 UTC m=+61.826777309"
Jul 02 13:15:50.002587 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.002385    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b9bbcdfbc-4gns6" podStartSLOduration=2.07754709 podStartE2EDuration="8.002367508s" totalImagesPullingTime="5.924820418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:50.00190353 +0000 UTC m=+61.857296436" watchObservedRunningTime="2026-07-02 13:15:50.002367508 +0000 UTC m=+61.857760370"
Jul 02 13:15:50.012738 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.012703    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-client-certs\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.012862 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.012750    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/894ee300-2398-4436-977c-82973679ed08-audit-log\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.012946 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.012886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.012946 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.012935    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-tls\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.013053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.012977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-client-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.013053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.013034    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-metrics-server-audit-profiles\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.013134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.013061    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdrw\" (UniqueName: \"kubernetes.io/projected/894ee300-2398-4436-977c-82973679ed08-kube-api-access-qsdrw\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.014267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.013958    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/894ee300-2398-4436-977c-82973679ed08-audit-log\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.014267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.014220    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.014446 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.014326    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/894ee300-2398-4436-977c-82973679ed08-metrics-server-audit-profiles\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.015642 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.015620    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-client-certs\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.017025 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.016984    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-client-ca-bundle\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.017259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.017237    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/894ee300-2398-4436-977c-82973679ed08-secret-metrics-server-tls\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.023081 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.023040    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdrw\" (UniqueName: \"kubernetes.io/projected/894ee300-2398-4436-977c-82973679ed08-kube-api-access-qsdrw\") pod \"metrics-server-b57f869fd-ppsvt\" (UID: \"894ee300-2398-4436-977c-82973679ed08\") " pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.145895 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.145802    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"]
Jul 02 13:15:50.148785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.148197    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.151296 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.151118    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8klgm\""
Jul 02 13:15:50.156415 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.155839    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 02 13:15:50.166582 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.166511    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"]
Jul 02 13:15:50.168395 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.168374    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:15:50.215715 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.215655    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.216067 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.216033    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb24x\" (UniqueName: \"kubernetes.io/projected/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-kube-api-access-qb24x\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.317522 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.317468    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.317672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.317566    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb24x\" (UniqueName: \"kubernetes.io/projected/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-kube-api-access-qb24x\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.317848 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:50.317823    2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 02 13:15:50.317991 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:15:50.317911    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert podName:beefa50a-cabb-4346-9ebb-e9a12b9e5acc nodeName:}" failed. No retries permitted until 2026-07-02 13:15:50.817890549 +0000 UTC m=+62.673283429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert") pod "monitoring-plugin-cd8656b8-4bkrh" (UID: "beefa50a-cabb-4346-9ebb-e9a12b9e5acc") : secret "monitoring-plugin-cert" not found
Jul 02 13:15:50.346730 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.346703    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb24x\" (UniqueName: \"kubernetes.io/projected/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-kube-api-access-qb24x\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.387928 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.387821    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b57f869fd-ppsvt"]
Jul 02 13:15:50.619646 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:50.619614    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894ee300_2398_4436_977c_82973679ed08.slice/crio-67b98078b68406e702f8cfd65252b6ace52296ab0ed959bfab30b3b988769013 WatchSource:0}: Error finding container 67b98078b68406e702f8cfd65252b6ace52296ab0ed959bfab30b3b988769013: Status 404 returned error can't find the container with id 67b98078b68406e702f8cfd65252b6ace52296ab0ed959bfab30b3b988769013
Jul 02 13:15:50.781379 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.781340    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nkxrg"
Jul 02 13:15:50.797017 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.796987    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"]
Jul 02 13:15:50.799605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.799585    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.802989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.802964    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 02 13:15:50.803100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.803023    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dmrvn\""
Jul 02 13:15:50.803372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.803170    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 02 13:15:50.803372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.803179    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 02 13:15:50.803372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.803261    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 02 13:15:50.803620 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.803589    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 02 13:15:50.811593 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.811569    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 02 13:15:50.823325 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.823293    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.825745 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.825720    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/beefa50a-cabb-4346-9ebb-e9a12b9e5acc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-4bkrh\" (UID: \"beefa50a-cabb-4346-9ebb-e9a12b9e5acc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:50.833784 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.833759    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"]
Jul 02 13:15:50.886093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.885943    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-26spv" event={"ID":"3cdca863-8d86-4583-8838-4607100cc261","Type":"ContainerStarted","Data":"4ad864fcf43f6b14eea59e2e6e591d513df8ad3bbf40837fb6f9ca9612c69ea4"}
Jul 02 13:15:50.888317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.887887    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt" event={"ID":"894ee300-2398-4436-977c-82973679ed08","Type":"ContainerStarted","Data":"67b98078b68406e702f8cfd65252b6ace52296ab0ed959bfab30b3b988769013"}
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924497    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-serving-certs-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924574    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tln9m\" (UniqueName: \"kubernetes.io/projected/2f793825-c7b1-4c69-9320-07d0ddd83b0f-kube-api-access-tln9m\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924648    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924732    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-metrics-client-ca\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924777    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-federate-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:50.925284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:50.924853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026368    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-metrics-client-ca\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-federate-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026830 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026830 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.026830 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026711    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-serving-certs-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.027018 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.026805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tln9m\" (UniqueName: \"kubernetes.io/projected/2f793825-c7b1-4c69-9320-07d0ddd83b0f-kube-api-access-tln9m\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.028101 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.028052    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-metrics-client-ca\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.028242 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.028221    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-serving-certs-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.028389 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.028365    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.031717 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.031666    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-federate-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.031927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.031839    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-telemeter-client-tls\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.032461 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.032420    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.034710 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.034664    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f793825-c7b1-4c69-9320-07d0ddd83b0f-secret-telemeter-client\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.047171 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.047147    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tln9m\" (UniqueName: \"kubernetes.io/projected/2f793825-c7b1-4c69-9320-07d0ddd83b0f-kube-api-access-tln9m\") pod \"telemeter-client-787f69c8cd-rxqmq\" (UID: \"2f793825-c7b1-4c69-9320-07d0ddd83b0f\") " pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.063909 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.063669    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:51.112219 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.112014    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"
Jul 02 13:15:51.579657 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.579603    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:51.592920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.592890    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.595888 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.595681    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 13:15:51.595888 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.595808    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 13:15:51.596057 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596043    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 13:15:51.596189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596170    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 13:15:51.596261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596199    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 13:15:51.596433 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596418    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 13:15:51.596489 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596434    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 13:15:51.596489 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596418    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 13:15:51.598152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596766    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 13:15:51.598152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596825    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2kkndvptn3i33\""
Jul 02 13:15:51.598152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.596991    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 13:15:51.598152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.597279    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 13:15:51.598152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.597679    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 13:15:51.598451 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.598226    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-44fvw\""
Jul 02 13:15:51.606012 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.605558    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:51.606108 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.606022    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 13:15:51.609361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.608411    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 13:15:51.735828 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.735766    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.735986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.735841    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.735986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.735926    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdl5w\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.735986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.735966    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736057    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736079    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736100    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736125    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736195    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736257    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736330    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736357    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736416    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736451    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736480    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736557    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.736697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.736587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837309 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837278    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837349    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837379    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837402    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837441    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.837718 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837473    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837944    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.837991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838034    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838066    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838128    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838155    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838155    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838194    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdl5w\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838226    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838319    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838346    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838370    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.838720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838393    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.839032 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.838862    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.839661 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.839336    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.839661 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.839364    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.840471 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.840163    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.840471 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.840461    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.841094    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.841139    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.841515    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.841814    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842093 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.842030    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842406 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.842160    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842767 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.842742    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842931 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.842915    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.842995 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.842950    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.843150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.843128    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.843480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.843466    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.844191 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.844174    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.850321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.850301    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdl5w\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w\") pod \"prometheus-k8s-0\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:51.892715 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.892658    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz" event={"ID":"0a472ad0-3f79-4836-b998-eaa7ddf9f939","Type":"ContainerStarted","Data":"cb509359aa5f8ef43200ba329df18333d60b828e57c9c2ac5ca436c049cbb66d"}
Jul 02 13:15:51.894161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.894141    2552 generic.go:356] "Generic (PLEG): container finished" podID="7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b" containerID="1b28b9f3f8de12bbec56ce07111a671cd62331825eb102879b26628220904cbe" exitCode=0
Jul 02 13:15:51.894233 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.894186    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmmhc" event={"ID":"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b","Type":"ContainerDied","Data":"1b28b9f3f8de12bbec56ce07111a671cd62331825eb102879b26628220904cbe"}
Jul 02 13:15:51.907856 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:51.907836    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:52.871665 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:52.871624    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:52.871844 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:52.871693    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:52.876931 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:52.876856    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:52.901869 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:52.901844    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:15:53.213052 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.212566    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"]
Jul 02 13:15:53.272902 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.272863    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:53.295782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.295753    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-787f69c8cd-rxqmq"]
Jul 02 13:15:53.299161 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:53.299121    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f793825_c7b1_4c69_9320_07d0ddd83b0f.slice/crio-87b4d71af1b6c62f757c1eaa6d5141b830af457e253fda08942e43304e073a09 WatchSource:0}: Error finding container 87b4d71af1b6c62f757c1eaa6d5141b830af457e253fda08942e43304e073a09: Status 404 returned error can't find the container with id 87b4d71af1b6c62f757c1eaa6d5141b830af457e253fda08942e43304e073a09
Jul 02 13:15:53.718271 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.718150    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:15:53.720533 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.720512    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.730143 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.730103    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:15:53.748216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.748160    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:15:53.861932 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.861900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.861945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.861996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.862023    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862222 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.862076    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862222 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.862105    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czfw\" (UniqueName: \"kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.862222 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.862141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.908265 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.907849    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz" event={"ID":"0a472ad0-3f79-4836-b998-eaa7ddf9f939","Type":"ContainerStarted","Data":"a8119ebb0b9475e49e6f7377259d929845d0d65c37978c9f833bc52fa605347b"}
Jul 02 13:15:53.908265 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.907951    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz" event={"ID":"0a472ad0-3f79-4836-b998-eaa7ddf9f939","Type":"ContainerStarted","Data":"dee53867e25163db7bf03b09fe997c58a143699e80251cd50d1118048f190dd7"}
Jul 02 13:15:53.911457 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.911158    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n" event={"ID":"e7a15b67-891f-40f8-a64b-30bb3a2ce04e","Type":"ContainerStarted","Data":"150d155ff154bc99153d2787947f80a2155fb068fd711f4c1b0ece93127a73f2"}
Jul 02 13:15:53.913008 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.912978    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh" event={"ID":"beefa50a-cabb-4346-9ebb-e9a12b9e5acc","Type":"ContainerStarted","Data":"561b8e25fad69fa7129a7547e8b3dd6f3e84b7716519168b49254415c4d52348"}
Jul 02 13:15:53.915412 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.915385    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"b3cf091cf5adfadbbf112d415f922eb52c51a88d6bb74918704d10c4a79130a6"}
Jul 02 13:15:53.915527 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.915453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"2f0a8ac9b84231ccf0612d153c0c043d8a4eda5220539709405c144086711a63"}
Jul 02 13:15:53.915527 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.915487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"5b4aae27cd846ad922ecd90c532f60eaeaf50e5bd248eb030ca490d44e941d42"}
Jul 02 13:15:53.917422 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.917398    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1" exitCode=0
Jul 02 13:15:53.917544 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.917489    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1"}
Jul 02 13:15:53.920384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.920300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmmhc" event={"ID":"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b","Type":"ContainerStarted","Data":"6b04c376c46c55736ef9eef5793f3931a4c2b8aa0505f76f483b32404e5297b4"}
Jul 02 13:15:53.920384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.920348    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mmmhc" event={"ID":"7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b","Type":"ContainerStarted","Data":"f9fad5e1f7c9b34b60d68c5fd437e56b2602525faa2c84ac2d1461a06d594414"}
Jul 02 13:15:53.922472 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.922449    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt" event={"ID":"894ee300-2398-4436-977c-82973679ed08","Type":"ContainerStarted","Data":"b09db77d6e84bf3bbc187341ecb9576525c0a33cf065786110de560be72809cb"}
Jul 02 13:15:53.924534 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.924507    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb" exitCode=0
Jul 02 13:15:53.924679 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.924587    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb"}
Jul 02 13:15:53.924679 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.924645    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"f5ff6c89112df75dda247965e34e0bd1aad6c1ce8db6944f36ccd26eb04feed4"}
Jul 02 13:15:53.927229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.927183    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq" event={"ID":"2f793825-c7b1-4c69-9320-07d0ddd83b0f","Type":"ContainerStarted","Data":"87b4d71af1b6c62f757c1eaa6d5141b830af457e253fda08942e43304e073a09"}
Jul 02 13:15:53.929189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.929140    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-ctlsz" podStartSLOduration=7.223337233 podStartE2EDuration="8.929123092s" totalImagesPullingTime="1.705785859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:53.927998061 +0000 UTC m=+65.783390967" watchObservedRunningTime="2026-07-02 13:15:53.929123092 +0000 UTC m=+65.784515978"
Jul 02 13:15:53.962762 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.962731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.962952 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.962782    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.962952 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.962828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963190 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963159    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963297 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963234    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963297 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963266    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czfw\" (UniqueName: \"kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963389 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963308    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963907 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963577    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.963907 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963616    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.964089 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.963966    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.964354 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.964314    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.965839 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.965802    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.966498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.966458    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:53.972489 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:53.972431    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czfw\" (UniqueName: \"kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw\") pod \"console-6dcccd4bc7-dsnwl\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") " pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:54.009597 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.009544    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-jc98n" podStartSLOduration=5.33570434 podStartE2EDuration="9.009525761s" totalImagesPullingTime="3.673821421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:54.008766702 +0000 UTC m=+65.864159587" watchObservedRunningTime="2026-07-02 13:15:54.009525761 +0000 UTC m=+65.864918647"
Jul 02 13:15:54.031081 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.031035    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mmmhc" podStartSLOduration=7.045954872 podStartE2EDuration="9.03102067s" totalImagesPullingTime="1.902304537s" totalInitContainerRuntime="82.761261ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:54.030108496 +0000 UTC m=+65.885501412" watchObservedRunningTime="2026-07-02 13:15:54.03102067 +0000 UTC m=+65.886413553"
Jul 02 13:15:54.031683 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.031623    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:15:54.057078 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.056995    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt" podStartSLOduration=2.6512149430000003 podStartE2EDuration="5.056947037s" totalImagesPullingTime="2.405732094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:54.051077783 +0000 UTC m=+65.906470707" watchObservedRunningTime="2026-07-02 13:15:54.056947037 +0000 UTC m=+65.912339919"
Jul 02 13:15:54.207459 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.207346    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:15:54.211447 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:54.211415    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1051ded_b604_45d9_b864_6c04ab0d5d9b.slice/crio-52daceed6a2b988e69e18101cf4b054939c5b254524bb4d7a4c8e71ddccb9503 WatchSource:0}: Error finding container 52daceed6a2b988e69e18101cf4b054939c5b254524bb4d7a4c8e71ddccb9503: Status 404 returned error can't find the container with id 52daceed6a2b988e69e18101cf4b054939c5b254524bb4d7a4c8e71ddccb9503
Jul 02 13:15:54.369499 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.369468    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:54.369694 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.369577    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:54.372607 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.372293    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:54.372607 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.372534    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:54.392891 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.392852    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86e3245-2705-4a72-b608-61bebd9eb17e-metrics-certs\") pod \"network-metrics-daemon-q9mnj\" (UID: \"c86e3245-2705-4a72-b608-61bebd9eb17e\") " pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:54.393236 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.393204    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f6ceabc-fa61-470e-8df7-db311b0433f6-original-pull-secret\") pod \"global-pull-secret-syncer-stzdv\" (UID: \"3f6ceabc-fa61-470e-8df7-db311b0433f6\") " pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:54.470468 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.470314    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:54.473454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.473425    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:54.483983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.483725    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:54.494594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.494537    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghff8\" (UniqueName: \"kubernetes.io/projected/6354334f-1524-4060-948b-c568d9382894-kube-api-access-ghff8\") pod \"network-check-target-w5957\" (UID: \"6354334f-1524-4060-948b-c568d9382894\") " pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:54.578791 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.578598    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lw488\""
Jul 02 13:15:54.581545 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.581524    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stzdv"
Jul 02 13:15:54.586177 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.586019    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:15:54.588461 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.588264    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zp2gh\""
Jul 02 13:15:54.596714 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.596561    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9mnj"
Jul 02 13:15:54.934661 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.934578    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcccd4bc7-dsnwl" event={"ID":"b1051ded-b604-45d9-b864-6c04ab0d5d9b","Type":"ContainerStarted","Data":"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"}
Jul 02 13:15:54.934661 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.934637    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcccd4bc7-dsnwl" event={"ID":"b1051ded-b604-45d9-b864-6c04ab0d5d9b","Type":"ContainerStarted","Data":"52daceed6a2b988e69e18101cf4b054939c5b254524bb4d7a4c8e71ddccb9503"}
Jul 02 13:15:54.955728 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:54.955299    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcccd4bc7-dsnwl" podStartSLOduration=1.9552774309999998 podStartE2EDuration="1.955277431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:54.953585714 +0000 UTC m=+66.808978631" watchObservedRunningTime="2026-07-02 13:15:54.955277431 +0000 UTC m=+66.810670308"
Jul 02 13:15:58.443668 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.443613    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5957"]
Jul 02 13:15:58.455674 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:58.455644    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6354334f_1524_4060_948b_c568d9382894.slice/crio-147dac0f6ea0e3162bd98c13cee4b0e1be494f03db537cea9c052a7806d6fc56 WatchSource:0}: Error finding container 147dac0f6ea0e3162bd98c13cee4b0e1be494f03db537cea9c052a7806d6fc56: Status 404 returned error can't find the container with id 147dac0f6ea0e3162bd98c13cee4b0e1be494f03db537cea9c052a7806d6fc56
Jul 02 13:15:58.469299 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.467851    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q9mnj"]
Jul 02 13:15:58.472973 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:15:58.472943    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86e3245_2705_4a72_b608_61bebd9eb17e.slice/crio-d3273c4f8f1be31f6de70d3108a27faf814c279170e8c25299f1d230c2ba66a2 WatchSource:0}: Error finding container d3273c4f8f1be31f6de70d3108a27faf814c279170e8c25299f1d230c2ba66a2: Status 404 returned error can't find the container with id d3273c4f8f1be31f6de70d3108a27faf814c279170e8c25299f1d230c2ba66a2
Jul 02 13:15:58.487129 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.484150    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stzdv"]
Jul 02 13:15:58.954563 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.954537    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh" event={"ID":"beefa50a-cabb-4346-9ebb-e9a12b9e5acc","Type":"ContainerStarted","Data":"eefe4782be65b9af63e6a45fc3337938f714c375ff8d7ebf7096b3976f24b9ce"}
Jul 02 13:15:58.954719 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.954701    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:58.957071 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.957044    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-26spv" event={"ID":"3cdca863-8d86-4583-8838-4607100cc261","Type":"ContainerStarted","Data":"c685971232385709cd9b96cf60017bf41b75323b29bc5dcd1bb357cbf87cddcd"}
Jul 02 13:15:58.960053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.960035    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh"
Jul 02 13:15:58.960924 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.960902    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"3a4c51cb541c338a5bbfae2f39b0a325dedbcf9b90a91ed1d878ba1e42cf4717"}
Jul 02 13:15:58.961008 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.960970    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"7071857e6cb6c761f15cadc31ce3d0b147b284a923bb571297a243c9dbf57773"}
Jul 02 13:15:58.961067 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.961021    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:58.961067 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.961062    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" event={"ID":"282fcef4-e4e9-4b69-8edf-62c585b3ffca","Type":"ContainerStarted","Data":"a789f8f8849433b7105eaeb6691087cba3eaa71b6381e2ee5e8cd388adcb9fd4"}
Jul 02 13:15:58.966217 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.966194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"}
Jul 02 13:15:58.966317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.966257    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"}
Jul 02 13:15:58.966317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.966302    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"}
Jul 02 13:15:58.967586 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.967561    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stzdv" event={"ID":"3f6ceabc-fa61-470e-8df7-db311b0433f6","Type":"ContainerStarted","Data":"8e01f84f6863a92b83239d6928029ae2ebb1932e146761a2345037f3d8c01206"}
Jul 02 13:15:58.968224 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.968172    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5"
Jul 02 13:15:58.970132 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.970110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"}
Jul 02 13:15:58.970291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.970254    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"}
Jul 02 13:15:58.970539 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.970401    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"}
Jul 02 13:15:58.972190 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.972150    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-4bkrh" podStartSLOduration=3.933713236 podStartE2EDuration="8.972134464s" totalImagesPullingTime="5.038421228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:58.970779821 +0000 UTC m=+70.826172707" watchObservedRunningTime="2026-07-02 13:15:58.972134464 +0000 UTC m=+70.827527348"
Jul 02 13:15:58.973667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.973630    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq" event={"ID":"2f793825-c7b1-4c69-9320-07d0ddd83b0f","Type":"ContainerStarted","Data":"ccfa31ade8c9380c98f7fe5fe2268583b3c20799b4d0f01ddebd59bc8ddd5191"}
Jul 02 13:15:58.973756 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.973707    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq" event={"ID":"2f793825-c7b1-4c69-9320-07d0ddd83b0f","Type":"ContainerStarted","Data":"03c64285e7bdc3b1ad4e8b0006c1b5debb4a96958638972088669edcb3d6fb53"}
Jul 02 13:15:58.973756 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.973740    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq" event={"ID":"2f793825-c7b1-4c69-9320-07d0ddd83b0f","Type":"ContainerStarted","Data":"220978886b555d074831f7f39201bcccf2a6aaed2e5dccef69cb9dfa48383de3"}
Jul 02 13:15:58.975115 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.975040    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5957" event={"ID":"6354334f-1524-4060-948b-c568d9382894","Type":"ContainerStarted","Data":"147dac0f6ea0e3162bd98c13cee4b0e1be494f03db537cea9c052a7806d6fc56"}
Jul 02 13:15:58.976455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.976439    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9mnj" event={"ID":"c86e3245-2705-4a72-b608-61bebd9eb17e","Type":"ContainerStarted","Data":"d3273c4f8f1be31f6de70d3108a27faf814c279170e8c25299f1d230c2ba66a2"}
Jul 02 13:15:58.990977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:58.990939    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-26spv" podStartSLOduration=1.896237167 podStartE2EDuration="17.990924715s" totalImagesPullingTime="16.094687548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:58.989396233 +0000 UTC m=+70.844789128" watchObservedRunningTime="2026-07-02 13:15:58.990924715 +0000 UTC m=+70.846317608"
Jul 02 13:15:59.017775 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.017727    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ccd878888-872p5" podStartSLOduration=2.332886816 podStartE2EDuration="11.017714042s" totalImagesPullingTime="8.684827226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:59.017415691 +0000 UTC m=+70.872808632" watchObservedRunningTime="2026-07-02 13:15:59.017714042 +0000 UTC m=+70.873107130"
Jul 02 13:15:59.061220 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.061162    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-787f69c8cd-rxqmq" podStartSLOduration=4.07023477 podStartE2EDuration="9.061141428s" totalImagesPullingTime="4.990906658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:59.059264344 +0000 UTC m=+70.914657262" watchObservedRunningTime="2026-07-02 13:15:59.061141428 +0000 UTC m=+70.916534315"
Jul 02 13:15:59.890025 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.889985    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-92cmb"
Jul 02 13:15:59.987893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.987843    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"}
Jul 02 13:15:59.988072 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.987956    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"}
Jul 02 13:15:59.988072 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.987999    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerStarted","Data":"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"}
Jul 02 13:15:59.993645 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.993599    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"}
Jul 02 13:15:59.993770 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.993691    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"}
Jul 02 13:15:59.993770 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:15:59.993736    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerStarted","Data":"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"}
Jul 02 13:16:00.082808 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:00.082633    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.9004565410000005 podStartE2EDuration="14.082609608s" totalImagesPullingTime="8.155374487s" totalInitContainerRuntime="26.77858ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:00.021386425 +0000 UTC m=+71.876779339" watchObservedRunningTime="2026-07-02 13:16:00.082609608 +0000 UTC m=+71.938002492"
Jul 02 13:16:00.083716 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:00.083377    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.597122555 podStartE2EDuration="9.083362975s" totalImagesPullingTime="4.455686446s" totalInitContainerRuntime="30.553974ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:00.080692632 +0000 UTC m=+71.936085542" watchObservedRunningTime="2026-07-02 13:16:00.083362975 +0000 UTC m=+71.938755860"
Jul 02 13:16:01.908150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:01.908110    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:02.560033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.559993    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:16:02.591840 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.591812    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:16:02.594340 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.594320    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.606637 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.606619    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:16:02.760572 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760538    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rb6l\" (UniqueName: \"kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760740 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760740 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760673    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760740 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760726    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760904 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760904 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.760904 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.760809    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862417 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862334    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rb6l\" (UniqueName: \"kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862417 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862420    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862484    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862531    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.862631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.862585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.863311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.863267    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.863426 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.863397    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.863502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.863452    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.863681 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.863659    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.865314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.865263    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.865508 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.865476    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.871674 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.871651    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rb6l\" (UniqueName: \"kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l\") pod \"console-6bc765bc4-pr8pj\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") " pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:02.905906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:02.905867    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:03.283861 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:03.283835    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:16:03.288192 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:16:03.288156    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54bb418_31a8_4957_aa3d_69f866eae3d9.slice/crio-77b3cda45749beef16f8a7143c844824cb566fb2761ce5ed84cca43536891f2d WatchSource:0}: Error finding container 77b3cda45749beef16f8a7143c844824cb566fb2761ce5ed84cca43536891f2d: Status 404 returned error can't find the container with id 77b3cda45749beef16f8a7143c844824cb566fb2761ce5ed84cca43536891f2d
Jul 02 13:16:04.013430 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.013395    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5957" event={"ID":"6354334f-1524-4060-948b-c568d9382894","Type":"ContainerStarted","Data":"e20ac64b2770d1de6b4471ab9da90ff858555ae47997e38546810c338b784acd"}
Jul 02 13:16:04.013595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.013521    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:16:04.015239 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.015215    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9mnj" event={"ID":"c86e3245-2705-4a72-b608-61bebd9eb17e","Type":"ContainerStarted","Data":"3547d28e4209b94d34f8fc1cfbb36a58b826921bb9418e9b2a7cac2a1f445739"}
Jul 02 13:16:04.015335 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.015260    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9mnj" event={"ID":"c86e3245-2705-4a72-b608-61bebd9eb17e","Type":"ContainerStarted","Data":"5e9eee30f0b650549e458051187dddacc690d14998c5360e28384811fa2d3fa2"}
Jul 02 13:16:04.016769 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.016746    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc765bc4-pr8pj" event={"ID":"b54bb418-31a8-4957-aa3d-69f866eae3d9","Type":"ContainerStarted","Data":"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"}
Jul 02 13:16:04.016893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.016782    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc765bc4-pr8pj" event={"ID":"b54bb418-31a8-4957-aa3d-69f866eae3d9","Type":"ContainerStarted","Data":"77b3cda45749beef16f8a7143c844824cb566fb2761ce5ed84cca43536891f2d"}
Jul 02 13:16:04.018142 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.018123    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stzdv" event={"ID":"3f6ceabc-fa61-470e-8df7-db311b0433f6","Type":"ContainerStarted","Data":"2c9a8189b3dd7d41e9172042ff9785f46488d9d493aedd777661df58e6358959"}
Jul 02 13:16:04.031450 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.031405    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w5957" podStartSLOduration=69.367508982 podStartE2EDuration="1m14.031392301s" totalImagesPullingTime="4.663883319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.029980539 +0000 UTC m=+75.885373424" watchObservedRunningTime="2026-07-02 13:16:04.031392301 +0000 UTC m=+75.886785330"
Jul 02 13:16:04.031908 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.031892    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:16:04.047183 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.047138    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-stzdv" podStartSLOduration=69.403651891 podStartE2EDuration="1m14.047125932s" totalImagesPullingTime="4.643474041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.04680807 +0000 UTC m=+75.902200955" watchObservedRunningTime="2026-07-02 13:16:04.047125932 +0000 UTC m=+75.902518812"
Jul 02 13:16:04.073391 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.073357    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc765bc4-pr8pj" podStartSLOduration=2.07334395 podStartE2EDuration="2.07334395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.072757616 +0000 UTC m=+75.928150527" watchObservedRunningTime="2026-07-02 13:16:04.07334395 +0000 UTC m=+75.928736894"
Jul 02 13:16:04.089276 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:04.089231    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q9mnj" podStartSLOduration=71.960072685 podStartE2EDuration="1m14.089216433s" totalImagesPullingTime="2.129143748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.088704522 +0000 UTC m=+75.944097439" watchObservedRunningTime="2026-07-02 13:16:04.089216433 +0000 UTC m=+75.944609318"
Jul 02 13:16:10.170555 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:10.170514    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:16:10.170970 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:10.170587    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:16:12.906840 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:12.906809    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:12.907317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:12.906853    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:12.911260 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:12.911226    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:13.060525 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:13.060502    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:16:13.109369 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:13.109343    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:16:27.582201 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.582162    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dcccd4bc7-dsnwl" podUID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" containerName="console" containerID="cri-o://c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504" gracePeriod=15
Jul 02 13:16:27.825250 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.825229    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:16:27.966431 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966405    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966478    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966506    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966578 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966564    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966702 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966625    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2czfw\" (UniqueName: \"kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966702 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966672    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966702 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966696    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert\") pod \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\" (UID: \"b1051ded-b604-45d9-b864-6c04ab0d5d9b\") "
Jul 02 13:16:27.966980 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966943    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config" (OuterVolumeSpecName: "console-config") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:27.967105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.966996    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:27.967105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.967041    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:27.967105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.967075    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:27.968890 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.968851    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:27.968991 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.968904    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:27.968991 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:27.968907    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw" (OuterVolumeSpecName: "kube-api-access-2czfw") pod "b1051ded-b604-45d9-b864-6c04ab0d5d9b" (UID: "b1051ded-b604-45d9-b864-6c04ab0d5d9b"). InnerVolumeSpecName "kube-api-access-2czfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:16:28.067405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067371    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czfw\" (UniqueName: \"kubernetes.io/projected/b1051ded-b604-45d9-b864-6c04ab0d5d9b-kube-api-access-2czfw\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067405    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-trusted-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067414    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067423    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-oauth-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067432    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-console-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067441    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-oauth-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.067564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.067449    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1051ded-b604-45d9-b864-6c04ab0d5d9b-service-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:28.124907 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.124860    2552 generic.go:356] "Generic (PLEG): container finished" podID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" containerID="c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504" exitCode=0
Jul 02 13:16:28.125011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.124922    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcccd4bc7-dsnwl" event={"ID":"b1051ded-b604-45d9-b864-6c04ab0d5d9b","Type":"ContainerDied","Data":"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"}
Jul 02 13:16:28.125011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.124948    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcccd4bc7-dsnwl"
Jul 02 13:16:28.125011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.124982    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcccd4bc7-dsnwl" event={"ID":"b1051ded-b604-45d9-b864-6c04ab0d5d9b","Type":"ContainerDied","Data":"52daceed6a2b988e69e18101cf4b054939c5b254524bb4d7a4c8e71ddccb9503"}
Jul 02 13:16:28.125011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.125009    2552 scope.go:117] "RemoveContainer" containerID="c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"
Jul 02 13:16:28.133729 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.133709    2552 scope.go:117] "RemoveContainer" containerID="c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"
Jul 02 13:16:28.134003 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:16:28.133984    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504\": container with ID starting with c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504 not found: ID does not exist" containerID="c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"
Jul 02 13:16:28.134060 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.134009    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504"} err="failed to get container status \"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504\": rpc error: code = NotFound desc = could not find container \"c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504\": container with ID starting with c5b83c5783fe0a459622d336b5c08ab5a70c152da6913945fb2c573ae1b69504 not found: ID does not exist"
Jul 02 13:16:28.154979 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.154960    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:16:28.161505 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:28.161484    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcccd4bc7-dsnwl"]
Jul 02 13:16:29.565909 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:29.565860    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" path="/var/lib/kubelet/pods/b1051ded-b604-45d9-b864-6c04ab0d5d9b/volumes"
Jul 02 13:16:30.176122 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:30.176092    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:16:30.179814 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:30.179787    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b57f869fd-ppsvt"
Jul 02 13:16:34.110337 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:34.110308    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/init-config-reloader/0.log"
Jul 02 13:16:34.312288 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:34.312247    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/alertmanager/0.log"
Jul 02 13:16:34.509608 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:34.509578    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/config-reloader/0.log"
Jul 02 13:16:34.709420 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:34.709386    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/kube-rbac-proxy-web/0.log"
Jul 02 13:16:34.911395 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:34.911316    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/kube-rbac-proxy/0.log"
Jul 02 13:16:35.025606 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:35.025581    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w5957"
Jul 02 13:16:35.110036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:35.110013    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/kube-rbac-proxy-metric/0.log"
Jul 02 13:16:35.309360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:35.309336    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2b0d777-c27a-4d59-9a07-5fc0a65c8655/prom-label-proxy/0.log"
Jul 02 13:16:35.709752 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:35.709683    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-state-metrics/0.log"
Jul 02 13:16:35.910808 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:35.910786    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-rbac-proxy-main/0.log"
Jul 02 13:16:36.110238 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:36.110214    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-rbac-proxy-self/0.log"
Jul 02 13:16:36.309574 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:36.309550    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b57f869fd-ppsvt_894ee300-2398-4436-977c-82973679ed08/metrics-server/0.log"
Jul 02 13:16:36.510955 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:36.510928    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-4bkrh_beefa50a-cabb-4346-9ebb-e9a12b9e5acc/monitoring-plugin/0.log"
Jul 02 13:16:37.309618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:37.309593    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/init-textfile/0.log"
Jul 02 13:16:37.509409 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:37.509382    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/node-exporter/0.log"
Jul 02 13:16:37.710463 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:37.710407    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/kube-rbac-proxy/0.log"
Jul 02 13:16:38.129433 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.129400    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5b9bbcdfbc-4gns6" podUID="1c22607d-25f7-4a17-9d52-00eab5364312" containerName="console" containerID="cri-o://bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380" gracePeriod=15
Jul 02 13:16:38.401033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.400975    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:16:38.450005 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.449981    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450148 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450013    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450148 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450040    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450148 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450107    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmpg7\" (UniqueName: \"kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450148 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450138    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450332 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450165    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert\") pod \"1c22607d-25f7-4a17-9d52-00eab5364312\" (UID: \"1c22607d-25f7-4a17-9d52-00eab5364312\") "
Jul 02 13:16:38.450674 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450557    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:38.450822 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450710    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:38.450822 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.450705    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config" (OuterVolumeSpecName: "console-config") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:16:38.455048 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.455002    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7" (OuterVolumeSpecName: "kube-api-access-cmpg7") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "kube-api-access-cmpg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:16:38.457973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.457935    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:38.458776 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.458752    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c22607d-25f7-4a17-9d52-00eab5364312" (UID: "1c22607d-25f7-4a17-9d52-00eab5364312"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:16:38.508974 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.508953    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/kube-rbac-proxy-main/0.log"
Jul 02 13:16:38.551654 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551633    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpg7\" (UniqueName: \"kubernetes.io/projected/1c22607d-25f7-4a17-9d52-00eab5364312-kube-api-access-cmpg7\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.551654 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551655    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-service-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.551774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551664    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-oauth-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.551774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551673    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.551774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551681    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c22607d-25f7-4a17-9d52-00eab5364312-console-oauth-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.551774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.551689    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c22607d-25f7-4a17-9d52-00eab5364312-console-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:16:38.710259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.710196    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/kube-rbac-proxy-self/0.log"
Jul 02 13:16:38.909381 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:38.909356    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/openshift-state-metrics/0.log"
Jul 02 13:16:39.112164 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.112126    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/init-config-reloader/0.log"
Jul 02 13:16:39.171687 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.171665    2552 generic.go:356] "Generic (PLEG): container finished" podID="1c22607d-25f7-4a17-9d52-00eab5364312" containerID="bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380" exitCode=0
Jul 02 13:16:39.171811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.171709    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9bbcdfbc-4gns6"
Jul 02 13:16:39.171811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.171756    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9bbcdfbc-4gns6" event={"ID":"1c22607d-25f7-4a17-9d52-00eab5364312","Type":"ContainerDied","Data":"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"}
Jul 02 13:16:39.171811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.171807    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9bbcdfbc-4gns6" event={"ID":"1c22607d-25f7-4a17-9d52-00eab5364312","Type":"ContainerDied","Data":"b844b8728d8b6b1fbf3c5f1b81031dafa25c0cffcd58baa414bb9626666ca69f"}
Jul 02 13:16:39.171931 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.171829    2552 scope.go:117] "RemoveContainer" containerID="bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"
Jul 02 13:16:39.180140 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.180126    2552 scope.go:117] "RemoveContainer" containerID="bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"
Jul 02 13:16:39.180396 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:16:39.180381    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380\": container with ID starting with bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380 not found: ID does not exist" containerID="bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"
Jul 02 13:16:39.180440 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.180404    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380"} err="failed to get container status \"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380\": rpc error: code = NotFound desc = could not find container \"bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380\": container with ID starting with bd2c6b88dcaee01c68bbee7850122d861982e176079facfa6650aefac5b0b380 not found: ID does not exist"
Jul 02 13:16:39.192799 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.192780    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:16:39.196204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.196184    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b9bbcdfbc-4gns6"]
Jul 02 13:16:39.309668 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.309650    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/prometheus/0.log"
Jul 02 13:16:39.510405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.510387    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/config-reloader/0.log"
Jul 02 13:16:39.565604 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.565571    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c22607d-25f7-4a17-9d52-00eab5364312" path="/var/lib/kubelet/pods/1c22607d-25f7-4a17-9d52-00eab5364312/volumes"
Jul 02 13:16:39.716325 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.716308    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/thanos-sidecar/0.log"
Jul 02 13:16:39.911163 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:39.911110    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/kube-rbac-proxy-web/0.log"
Jul 02 13:16:40.111291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:40.111267    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/kube-rbac-proxy/0.log"
Jul 02 13:16:40.310547 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:40.310528    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_573ba3f6-9083-4e6c-8436-9f290a1f1a88/kube-rbac-proxy-thanos/0.log"
Jul 02 13:16:41.112136 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:41.112096    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/telemeter-client/0.log"
Jul 02 13:16:41.311315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:41.311286    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/reload/0.log"
Jul 02 13:16:41.509514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:41.509493    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/kube-rbac-proxy/0.log"
Jul 02 13:16:41.710334 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:41.710301    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/thanos-query/0.log"
Jul 02 13:16:41.910864 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:41.910790    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-web/0.log"
Jul 02 13:16:42.109959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:42.109938    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy/0.log"
Jul 02 13:16:42.310045 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:42.310026    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/prom-label-proxy/0.log"
Jul 02 13:16:42.514237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:42.514209    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-rules/0.log"
Jul 02 13:16:42.708837 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:42.708774    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-metrics/0.log"
Jul 02 13:16:43.709805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:43.709773    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bc765bc4-pr8pj_b54bb418-31a8-4957-aa3d-69f866eae3d9/console/0.log"
Jul 02 13:16:44.309939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:44.309898    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w57dk_720c4f68-b38c-4924-9d7f-f027bf16cdd8/serve-healthcheck-canary/0.log"
Jul 02 13:16:51.908815 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:51.908786    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:51.929268 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:51.929247    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:52.245603 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:16:52.245579    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:05.791746 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.791705    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:05.792330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792074    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="alertmanager" containerID="cri-o://68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420" gracePeriod=120
Jul 02 13:17:05.792330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792127    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-metric" containerID="cri-o://75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150" gracePeriod=120
Jul 02 13:17:05.792330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792163    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy" containerID="cri-o://47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212" gracePeriod=120
Jul 02 13:17:05.792330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792185    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="config-reloader" containerID="cri-o://854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879" gracePeriod=120
Jul 02 13:17:05.792330 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792186    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="prom-label-proxy" containerID="cri-o://e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1" gracePeriod=120
Jul 02 13:17:05.792694 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:05.792145    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-web" containerID="cri-o://63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3" gracePeriod=120
Jul 02 13:17:06.286110 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286081    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1" exitCode=0
Jul 02 13:17:06.286110 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286105    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150" exitCode=0
Jul 02 13:17:06.286110 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286112    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212" exitCode=0
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286119    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879" exitCode=0
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286125    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420" exitCode=0
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286148    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"}
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286210    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"}
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286236    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"}
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286258    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"}
Jul 02 13:17:06.286307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:06.286282    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"}
Jul 02 13:17:07.025253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.025231    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.169941 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.169833    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.169941 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.169908    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.169941 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.169941    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.169962    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.169984    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170016    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170263    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170294    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mq7n\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170321    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170345    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:07.170404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170356    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170620 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170436    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170620 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170466    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca\") pod \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\" (UID: \"a2b0d777-c27a-4d59-9a07-5fc0a65c8655\") "
Jul 02 13:17:07.170785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.170768    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.171161 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.171140    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:07.171372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.171351    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:07.172892 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.172812    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out" (OuterVolumeSpecName: "config-out") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:07.173346 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.173062    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.173346 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.173181    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.173518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.173496    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n" (OuterVolumeSpecName: "kube-api-access-9mq7n") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "kube-api-access-9mq7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:07.173665 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.173593    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.174139 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.174108    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.174671 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.174643    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume" (OuterVolumeSpecName: "config-volume") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.175057 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.175032    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:07.177756 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.177737    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.183954 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.183932    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config" (OuterVolumeSpecName: "web-config") pod "a2b0d777-c27a-4d59-9a07-5fc0a65c8655" (UID: "a2b0d777-c27a-4d59-9a07-5fc0a65c8655"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:07.272129 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272101    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mq7n\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-kube-api-access-9mq7n\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272129 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272128    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-tls-assets\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272138    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-web-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272146    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-alertmanager-main-db\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272155    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-metrics-client-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272163    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-out\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272172    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272181    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-main-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272190    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-cluster-tls-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272198    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272206    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-config-volume\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.272284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.272214    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2b0d777-c27a-4d59-9a07-5fc0a65c8655-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:07.293062 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.293037    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerID="63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3" exitCode=0
Jul 02 13:17:07.293152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.293112    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"}
Jul 02 13:17:07.293152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.293134    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.293228 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.293209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2b0d777-c27a-4d59-9a07-5fc0a65c8655","Type":"ContainerDied","Data":"0505fe88c72878f3308987a110ad8edaeb5a27779ed411f4303ce259dc4da182"}
Jul 02 13:17:07.293257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.293243    2552 scope.go:117] "RemoveContainer" containerID="e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"
Jul 02 13:17:07.301080 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.301065    2552 scope.go:117] "RemoveContainer" containerID="75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"
Jul 02 13:17:07.307581 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.307566    2552 scope.go:117] "RemoveContainer" containerID="47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"
Jul 02 13:17:07.314033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.314017    2552 scope.go:117] "RemoveContainer" containerID="63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"
Jul 02 13:17:07.317674 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.317654    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:07.321881 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.321851    2552 scope.go:117] "RemoveContainer" containerID="854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"
Jul 02 13:17:07.322152 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.322128    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:07.328471 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.328453    2552 scope.go:117] "RemoveContainer" containerID="68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"
Jul 02 13:17:07.334933 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.334918    2552 scope.go:117] "RemoveContainer" containerID="38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1"
Jul 02 13:17:07.341270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341256    2552 scope.go:117] "RemoveContainer" containerID="e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"
Jul 02 13:17:07.341487 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.341470    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1\": container with ID starting with e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1 not found: ID does not exist" containerID="e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"
Jul 02 13:17:07.341530 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341495    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1"} err="failed to get container status \"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1\": rpc error: code = NotFound desc = could not find container \"e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1\": container with ID starting with e150ea443b728cbf5a1899c3265357bc683981dc0352e6000a56c63da81294c1 not found: ID does not exist"
Jul 02 13:17:07.341530 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341517    2552 scope.go:117] "RemoveContainer" containerID="75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"
Jul 02 13:17:07.341713 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.341697    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150\": container with ID starting with 75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150 not found: ID does not exist" containerID="75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"
Jul 02 13:17:07.341751 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341719    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150"} err="failed to get container status \"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150\": rpc error: code = NotFound desc = could not find container \"75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150\": container with ID starting with 75a0b15b957b4f7da335d5304ca31bff42463e293353bf624ca0ef4f94370150 not found: ID does not exist"
Jul 02 13:17:07.341751 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341734    2552 scope.go:117] "RemoveContainer" containerID="47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"
Jul 02 13:17:07.341968 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.341953    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212\": container with ID starting with 47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212 not found: ID does not exist" containerID="47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"
Jul 02 13:17:07.342016 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341972    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212"} err="failed to get container status \"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212\": rpc error: code = NotFound desc = could not find container \"47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212\": container with ID starting with 47f88af743e8d29e5bf7e78b0db72099b5a82f44033efb64568ad19390deb212 not found: ID does not exist"
Jul 02 13:17:07.342016 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.341984    2552 scope.go:117] "RemoveContainer" containerID="63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"
Jul 02 13:17:07.342182 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.342165    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3\": container with ID starting with 63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3 not found: ID does not exist" containerID="63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"
Jul 02 13:17:07.342265 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342184    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3"} err="failed to get container status \"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3\": rpc error: code = NotFound desc = could not find container \"63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3\": container with ID starting with 63296235fcf40a00471ab72ac6cd09dc6b23b1113a2ff54d529d4f931d7644e3 not found: ID does not exist"
Jul 02 13:17:07.342265 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342196    2552 scope.go:117] "RemoveContainer" containerID="854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"
Jul 02 13:17:07.342467 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.342449    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879\": container with ID starting with 854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879 not found: ID does not exist" containerID="854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"
Jul 02 13:17:07.342532 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342474    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879"} err="failed to get container status \"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879\": rpc error: code = NotFound desc = could not find container \"854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879\": container with ID starting with 854814a4b6a3e7024c01766dee313910a51f1071a9a01eb5f44cf2bd92486879 not found: ID does not exist"
Jul 02 13:17:07.342532 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342493    2552 scope.go:117] "RemoveContainer" containerID="68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"
Jul 02 13:17:07.342778 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.342731    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420\": container with ID starting with 68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420 not found: ID does not exist" containerID="68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"
Jul 02 13:17:07.342865 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342783    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420"} err="failed to get container status \"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420\": rpc error: code = NotFound desc = could not find container \"68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420\": container with ID starting with 68eba09077ff128a8eb1f54e2750cc4bcf82548599951872e7776e1bf7c67420 not found: ID does not exist"
Jul 02 13:17:07.342865 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.342803    2552 scope.go:117] "RemoveContainer" containerID="38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1"
Jul 02 13:17:07.343141 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:07.343122    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1\": container with ID starting with 38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1 not found: ID does not exist" containerID="38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1"
Jul 02 13:17:07.343237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.343145    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1"} err="failed to get container status \"38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1\": rpc error: code = NotFound desc = could not find container \"38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1\": container with ID starting with 38a6a36a8de33d845a28666eade1dee6945dd9c5b24c8c1153c044babba33eb1 not found: ID does not exist"
Jul 02 13:17:07.360703 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.360683    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:07.360975 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.360963    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="config-reloader"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.360978    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="config-reloader"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.360991    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="init-config-reloader"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.360997    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="init-config-reloader"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361004    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="prom-label-proxy"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361009    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="prom-label-proxy"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361017    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:07.361024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361022    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361031    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-web"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361036    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-web"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361043    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c22607d-25f7-4a17-9d52-00eab5364312" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361048    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c22607d-25f7-4a17-9d52-00eab5364312" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361053    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="alertmanager"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361058    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="alertmanager"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361068    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361072    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361078    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361083    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361140    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1051ded-b604-45d9-b864-6c04ab0d5d9b" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361147    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="alertmanager"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361151    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="config-reloader"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361158    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c22607d-25f7-4a17-9d52-00eab5364312" containerName="console"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361163    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361168    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-web"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361173    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:07.361226 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.361178    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" containerName="prom-label-proxy"
Jul 02 13:17:07.365479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.365465    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.368165 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368139    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 13:17:07.368269 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368164    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 13:17:07.368432 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368419    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 13:17:07.368500 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368475    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 13:17:07.368564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368507    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-px427\""
Jul 02 13:17:07.368632 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368513    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 13:17:07.368798 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368777    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 13:17:07.368984 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.368971    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 13:17:07.369037 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.369020    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 13:17:07.374426 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.374409    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 13:17:07.381647 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.381624    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:07.473487 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473453    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473487 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473500    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-web-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473594    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473623    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-out\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473642    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473684    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473716    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473731    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473751    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473780    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2n5\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-kube-api-access-4h2n5\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.473950 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.473814    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.564951 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.564926    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b0d777-c27a-4d59-9a07-5fc0a65c8655" path="/var/lib/kubelet/pods/a2b0d777-c27a-4d59-9a07-5fc0a65c8655/volumes"
Jul 02 13:17:07.575136 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575120    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575192 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575148    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575192 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575173    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-web-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575266 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575214    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575294    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-out\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.575496 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.575402    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576074    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576169    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576195    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2n5\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-kube-api-access-4h2n5\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576200    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576308    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.576540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.576421    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ee3f7e-3228-45c9-ac95-b4a9416914e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.577995 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.577950    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-out\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.578131 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.578116    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-web-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.578268 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.578249    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.578362 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.578341    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.578773 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.578753    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.579131 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.579110    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.579301 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.579283    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.579564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.579545    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.579718 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.579703    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53ee3f7e-3228-45c9-ac95-b4a9416914e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.584545 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.584529    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2n5\" (UniqueName: \"kubernetes.io/projected/53ee3f7e-3228-45c9-ac95-b4a9416914e8-kube-api-access-4h2n5\") pod \"alertmanager-main-0\" (UID: \"53ee3f7e-3228-45c9-ac95-b4a9416914e8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.675329 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.675304    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:07.810125 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:07.810049    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:07.812155 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:17:07.812128    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ee3f7e_3228_45c9_ac95_b4a9416914e8.slice/crio-42a8d528488600421ede1545c6fc12c3cd221ca9e53c0163e55a17c4aab850bd WatchSource:0}: Error finding container 42a8d528488600421ede1545c6fc12c3cd221ca9e53c0163e55a17c4aab850bd: Status 404 returned error can't find the container with id 42a8d528488600421ede1545c6fc12c3cd221ca9e53c0163e55a17c4aab850bd
Jul 02 13:17:08.298416 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:08.298383    2552 generic.go:356] "Generic (PLEG): container finished" podID="53ee3f7e-3228-45c9-ac95-b4a9416914e8" containerID="9e23cec62b8a51a948b1445a9df4c6900709395b1d8e3b7b960c82bba889e39e" exitCode=0
Jul 02 13:17:08.298784 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:08.298468    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerDied","Data":"9e23cec62b8a51a948b1445a9df4c6900709395b1d8e3b7b960c82bba889e39e"}
Jul 02 13:17:08.298784 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:08.298534    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"42a8d528488600421ede1545c6fc12c3cd221ca9e53c0163e55a17c4aab850bd"}
Jul 02 13:17:09.308353 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"da8d83d60a72119019e5507562d68110231c5f9398ce218ef54c8bc146c4e91c"}
Jul 02 13:17:09.308782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308432    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"8b29e83d0612eb49c7517f49b1a7b2f3226cc7a5e0ce160a38edabcd981a8df1"}
Jul 02 13:17:09.308782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308479    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"9b519414fbf386a68bd0075b6a4d805293ae9a1caf41b8d45af82e4b4163043c"}
Jul 02 13:17:09.308782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308523    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"735e5778c1e143799862696049d78e0b955a2010ef137c8f30c1c507beaee05f"}
Jul 02 13:17:09.308782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308558    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"68bc3d5be525cb0ef27f64686109c6adea6d94ace4b4486cc7784fcf9e1e677d"}
Jul 02 13:17:09.308782 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.308596    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53ee3f7e-3228-45c9-ac95-b4a9416914e8","Type":"ContainerStarted","Data":"02b468a5d881a1d2920b848e9aba5006c69002750ecc5848df4f2198ee14aff6"}
Jul 02 13:17:09.338499 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:09.338449    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.324252882 podStartE2EDuration="2.338435522s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.18264ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:09.335753997 +0000 UTC m=+141.191146894" watchObservedRunningTime="2026-07-02 13:17:09.338435522 +0000 UTC m=+141.193828404"
Jul 02 13:17:10.022888 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.022852    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:10.023178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023146    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="prometheus" containerID="cri-o://9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c" gracePeriod=600
Jul 02 13:17:10.023178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023177    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09" gracePeriod=600
Jul 02 13:17:10.023383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023182    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="thanos-sidecar" containerID="cri-o://5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80" gracePeriod=600
Jul 02 13:17:10.023383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023228    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="config-reloader" containerID="cri-o://2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b" gracePeriod=600
Jul 02 13:17:10.023383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023152    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy" containerID="cri-o://b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053" gracePeriod=600
Jul 02 13:17:10.023383 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.023191    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-web" containerID="cri-o://b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5" gracePeriod=600
Jul 02 13:17:10.316316 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316254    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09" exitCode=0
Jul 02 13:17:10.316316 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316275    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053" exitCode=0
Jul 02 13:17:10.316316 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316281    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80" exitCode=0
Jul 02 13:17:10.316316 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316286    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b" exitCode=0
Jul 02 13:17:10.316316 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316291    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c" exitCode=0
Jul 02 13:17:10.316741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316320    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"}
Jul 02 13:17:10.316741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316380    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"}
Jul 02 13:17:10.316741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316417    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"}
Jul 02 13:17:10.316741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316442    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"}
Jul 02 13:17:10.316741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:10.316465    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"}
Jul 02 13:17:11.263512 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.263483    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.312007 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.311985    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312022    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312064    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312100    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312273 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312141    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312273 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312169    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312358 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312271    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312358 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312321    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312355    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312393    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312426    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312553 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312456    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312553 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312481    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312553 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312520    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312553 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312548    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312601    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdl5w\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312628    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.312667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.312655    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.313389 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.313362    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.313791 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.313526    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.313791 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.313579    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.313791 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.313626    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2\") pod \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\" (UID: \"573ba3f6-9083-4e6c-8436-9f290a1f1a88\") "
Jul 02 13:17:11.314018 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.313966    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.314454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.314302    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.314454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.314331    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-metrics-client-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.314454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.314347    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.315090 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.314999    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.315679 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.315655    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.316266 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.316213    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.316359 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.316278    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.317408 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.317373    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:11.317680 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.317632    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:11.318205 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318167    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.318396 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318337    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.318801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318578    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.318801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318671    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.318801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318730    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.319027 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.318910    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out" (OuterVolumeSpecName: "config-out") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:11.319749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.319704    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w" (OuterVolumeSpecName: "kube-api-access-xdl5w") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "kube-api-access-xdl5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:11.320033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.319998    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config" (OuterVolumeSpecName: "config") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.320565 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.320541    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.321637 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.321131    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.326703 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.326678    2552 generic.go:356] "Generic (PLEG): container finished" podID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerID="b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5" exitCode=0
Jul 02 13:17:11.326822 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.326804    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"}
Jul 02 13:17:11.326889 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.326842    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.326969 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.326951    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"573ba3f6-9083-4e6c-8436-9f290a1f1a88","Type":"ContainerDied","Data":"f5ff6c89112df75dda247965e34e0bd1aad6c1ce8db6944f36ccd26eb04feed4"}
Jul 02 13:17:11.327018 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.327006    2552 scope.go:117] "RemoveContainer" containerID="dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"
Jul 02 13:17:11.329092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.329067    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config" (OuterVolumeSpecName: "web-config") pod "573ba3f6-9083-4e6c-8436-9f290a1f1a88" (UID: "573ba3f6-9083-4e6c-8436-9f290a1f1a88"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.336810 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.336792    2552 scope.go:117] "RemoveContainer" containerID="b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"
Jul 02 13:17:11.344369 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.344353    2552 scope.go:117] "RemoveContainer" containerID="b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"
Jul 02 13:17:11.350527 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.350513    2552 scope.go:117] "RemoveContainer" containerID="5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"
Jul 02 13:17:11.356533 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.356518    2552 scope.go:117] "RemoveContainer" containerID="2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"
Jul 02 13:17:11.363955 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.363935    2552 scope.go:117] "RemoveContainer" containerID="9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"
Jul 02 13:17:11.370721 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.370706    2552 scope.go:117] "RemoveContainer" containerID="4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb"
Jul 02 13:17:11.376765 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.376751    2552 scope.go:117] "RemoveContainer" containerID="dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"
Jul 02 13:17:11.377000 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.376978    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09\": container with ID starting with dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09 not found: ID does not exist" containerID="dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"
Jul 02 13:17:11.377048 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377010    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09"} err="failed to get container status \"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09\": rpc error: code = NotFound desc = could not find container \"dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09\": container with ID starting with dbe93cdcac38d9c7d484e80c93030a98a7dc820b18178a01228bd8e8bc538a09 not found: ID does not exist"
Jul 02 13:17:11.377048 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377028    2552 scope.go:117] "RemoveContainer" containerID="b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"
Jul 02 13:17:11.377239 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.377218    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053\": container with ID starting with b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053 not found: ID does not exist" containerID="b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"
Jul 02 13:17:11.377280 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377247    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053"} err="failed to get container status \"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053\": rpc error: code = NotFound desc = could not find container \"b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053\": container with ID starting with b4071f631af51f400b203d583b74c4f16dd2cd4dbe34ef33f21981617b9df053 not found: ID does not exist"
Jul 02 13:17:11.377280 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377262    2552 scope.go:117] "RemoveContainer" containerID="b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"
Jul 02 13:17:11.377481 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.377467    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5\": container with ID starting with b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5 not found: ID does not exist" containerID="b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"
Jul 02 13:17:11.377518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377486    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5"} err="failed to get container status \"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5\": rpc error: code = NotFound desc = could not find container \"b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5\": container with ID starting with b9d836dcfed631707d88951f27a8cb908f4fa907b0e63280567b9ac02d7ea3a5 not found: ID does not exist"
Jul 02 13:17:11.377518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377498    2552 scope.go:117] "RemoveContainer" containerID="5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"
Jul 02 13:17:11.377673 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.377658    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80\": container with ID starting with 5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80 not found: ID does not exist" containerID="5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"
Jul 02 13:17:11.377723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377677    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80"} err="failed to get container status \"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80\": rpc error: code = NotFound desc = could not find container \"5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80\": container with ID starting with 5cb43b81795d3c892dcb99f5a5ca7316d96090273fdc3a6e06dcc9726b631a80 not found: ID does not exist"
Jul 02 13:17:11.377723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377688    2552 scope.go:117] "RemoveContainer" containerID="2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"
Jul 02 13:17:11.377935 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.377920    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b\": container with ID starting with 2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b not found: ID does not exist" containerID="2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"
Jul 02 13:17:11.377977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377939    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b"} err="failed to get container status \"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b\": rpc error: code = NotFound desc = could not find container \"2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b\": container with ID starting with 2a45065fb8ac98c048da5b4f44d89830053e05e722bcf577329a4e9e9cc8fa2b not found: ID does not exist"
Jul 02 13:17:11.377977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.377951    2552 scope.go:117] "RemoveContainer" containerID="9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"
Jul 02 13:17:11.378169 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.378155    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c\": container with ID starting with 9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c not found: ID does not exist" containerID="9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"
Jul 02 13:17:11.378204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.378175    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c"} err="failed to get container status \"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c\": rpc error: code = NotFound desc = could not find container \"9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c\": container with ID starting with 9ab0d9c6370adfa9be7ff42103c2aa8fbdc2819fb1f48b333693318a74c2125c not found: ID does not exist"
Jul 02 13:17:11.378204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.378191    2552 scope.go:117] "RemoveContainer" containerID="4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb"
Jul 02 13:17:11.378409 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:11.378395    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb\": container with ID starting with 4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb not found: ID does not exist" containerID="4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb"
Jul 02 13:17:11.378456 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.378412    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb"} err="failed to get container status \"4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb\": rpc error: code = NotFound desc = could not find container \"4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb\": container with ID starting with 4214786a7d9f3ffe23461f31f36f32d8ea17b05dc1caef2cc9db6574868fd2cb not found: ID does not exist"
Jul 02 13:17:11.414959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.414938    2552 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.414967    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-tls-assets\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.414976    2552 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-kube-rbac-proxy\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.414985    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.414994    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415002    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415010    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415019    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config-out\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415027    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415036    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdl5w\" (UniqueName: \"kubernetes.io/projected/573ba3f6-9083-4e6c-8436-9f290a1f1a88-kube-api-access-xdl5w\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415044    2552 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415053    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-db\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415062    2552 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-metrics-client-certs\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415071    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/573ba3f6-9083-4e6c-8436-9f290a1f1a88-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415079    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-web-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415087    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.415352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.415094    2552 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/573ba3f6-9083-4e6c-8436-9f290a1f1a88-secret-grpc-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.645562 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.645497    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:11.650959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.650936    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:11.680360 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680335    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:11.680610 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680599    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="prometheus"
Jul 02 13:17:11.680650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680611    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="prometheus"
Jul 02 13:17:11.680650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680621    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="config-reloader"
Jul 02 13:17:11.680650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680626    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="config-reloader"
Jul 02 13:17:11.680650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680640    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-web"
Jul 02 13:17:11.680650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680645    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-web"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680653    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680658    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680665    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="init-config-reloader"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680670    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="init-config-reloader"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680682    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="thanos-sidecar"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680687    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="thanos-sidecar"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680695    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680699    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680743    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680751    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="config-reloader"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680757    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="prometheus"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680762    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680767    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="thanos-sidecar"
Jul 02 13:17:11.680794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.680773    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" containerName="kube-rbac-proxy-web"
Jul 02 13:17:11.684415 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.684398    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.687746 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.687726    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 13:17:11.687847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.687732    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 13:17:11.687847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.687732    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 13:17:11.688513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688462    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 13:17:11.688513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688471    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 13:17:11.688513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688466    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 13:17:11.688788 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688774    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-44fvw\""
Jul 02 13:17:11.688854 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688834    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 13:17:11.688936 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688925    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 13:17:11.688992 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.688978    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 13:17:11.689071 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.689055    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 13:17:11.689124 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.689078    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 13:17:11.689124 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.689086    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 13:17:11.690917 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.690897    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 13:17:11.691439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.691407    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2kkndvptn3i33\""
Jul 02 13:17:11.695559 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.695541    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 13:17:11.702825 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.702807    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:11.717417 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717428    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717455    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717510    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717538    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717560    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717768 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717616    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717768 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717643    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717768 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717668    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717768 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717767    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717790    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717812    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717831    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgx7\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-kube-api-access-mbgx7\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717854    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.717896 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717882    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.718092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717937    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.718092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717972    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.718092 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.717993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819080 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819052    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819080 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819084    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819122    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819155    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819179    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819261    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgx7\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-kube-api-access-mbgx7\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819315 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819308    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819362    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819402    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819472    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819500    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819540    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819600    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.819747 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.820185 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.819759    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.820185 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.820034    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.820185 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.820085    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.820337 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.820255    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.821062 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.820735    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.821062 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.820753    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.822783 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.822756    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.822945 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.822925    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823140 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823115    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823325 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823306    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823521 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823465    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b5c3146-9b10-447b-ae11-b9c8c80a783d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823591    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823922 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823902    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.823969 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.823902    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.824397 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.824376    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.824455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.824383    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.825157 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.825140    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.825374 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.825355    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.825413 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.825364    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b5c3146-9b10-447b-ae11-b9c8c80a783d-config\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.829070 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.829054    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgx7\" (UniqueName: \"kubernetes.io/projected/3b5c3146-9b10-447b-ae11-b9c8c80a783d-kube-api-access-mbgx7\") pod \"prometheus-k8s-0\" (UID: \"3b5c3146-9b10-447b-ae11-b9c8c80a783d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:11.994300 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:11.994273    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:12.131842 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:12.131817    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:12.133451 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:17:12.133424    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5c3146_9b10_447b_ae11_b9c8c80a783d.slice/crio-9879f705651083cfc9f420ace690c23428a21835d11862ee25b0c68a25b3789d WatchSource:0}: Error finding container 9879f705651083cfc9f420ace690c23428a21835d11862ee25b0c68a25b3789d: Status 404 returned error can't find the container with id 9879f705651083cfc9f420ace690c23428a21835d11862ee25b0c68a25b3789d
Jul 02 13:17:12.331711 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:12.331656    2552 generic.go:356] "Generic (PLEG): container finished" podID="3b5c3146-9b10-447b-ae11-b9c8c80a783d" containerID="c3fb52b2aa3c91d34dcf1f486f05f5d0fc99e899af35b8b60dd8c22807a46797" exitCode=0
Jul 02 13:17:12.332061 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:12.331738    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerDied","Data":"c3fb52b2aa3c91d34dcf1f486f05f5d0fc99e899af35b8b60dd8c22807a46797"}
Jul 02 13:17:12.332061 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:12.331815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"9879f705651083cfc9f420ace690c23428a21835d11862ee25b0c68a25b3789d"}
Jul 02 13:17:13.340527 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340493    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"3c4aa1366e5fe1a23b4e3274b6d37dff27ccc2f190961cdca8b0700ecef9ef61"}
Jul 02 13:17:13.340925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340556    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"6279b8d8ca168727e7f0425b84c10690e9f963514935988f45f6852ddd220b20"}
Jul 02 13:17:13.340925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340584    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"696b99249d4a65ecabcc5293cbd17e78481329e8a78c3923763347d382657b82"}
Jul 02 13:17:13.340925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340613    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"0c59259d6e600da4f500e666c56e0948f16491dd0ac42c1073533436a21ea965"}
Jul 02 13:17:13.340925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340638    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"e4266c2956e12dcc47d424b8402f047afa48a606497ff174216ea400181da06a"}
Jul 02 13:17:13.340925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.340661    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b5c3146-9b10-447b-ae11-b9c8c80a783d","Type":"ContainerStarted","Data":"6f7350901cf9995576f3a4b5de23042b1a41ebfbf4ed6fe213d0438b943aa051"}
Jul 02 13:17:13.374860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.374763    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.356176052 podStartE2EDuration="2.374746385s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.570333ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:13.373406762 +0000 UTC m=+145.228799666" watchObservedRunningTime="2026-07-02 13:17:13.374746385 +0000 UTC m=+145.230139268"
Jul 02 13:17:13.565940 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:13.565915    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573ba3f6-9083-4e6c-8436-9f290a1f1a88" path="/var/lib/kubelet/pods/573ba3f6-9083-4e6c-8436-9f290a1f1a88/volumes"
Jul 02 13:17:16.994834 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:16.994803    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:22.906066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:22.906034    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:17:22.909207 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:22.909182    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:22.921185 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:22.921163    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:17:23.000308 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jtr\" (UniqueName: \"kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000433 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000312    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000433 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000329    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000433 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000361    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000567 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000438    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000567 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000480    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.000567 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.000542    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101412 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101380    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101412 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101416    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101438    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101478    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jtr\" (UniqueName: \"kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101494    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101508    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.101531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.101526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.102165 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.102144    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.102258 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.102198    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.102328 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.102263    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.102524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.102496    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.103826 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.103807    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.103925 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.103912    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.109203 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.109180    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jtr\" (UniqueName: \"kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr\") pod \"console-b67948f45-sgmdm\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") " pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.217997 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.217980    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:23.346795 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.346661    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:17:23.349626 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:17:23.349600    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0189327b_65b9_48e5_9886_49ad5f49611b.slice/crio-a9f8433f6e165eb03b36a1d73980c1fe8b7c753cf409e18f54c308af003b7f43 WatchSource:0}: Error finding container a9f8433f6e165eb03b36a1d73980c1fe8b7c753cf409e18f54c308af003b7f43: Status 404 returned error can't find the container with id a9f8433f6e165eb03b36a1d73980c1fe8b7c753cf409e18f54c308af003b7f43
Jul 02 13:17:23.383034 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:23.382994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b67948f45-sgmdm" event={"ID":"0189327b-65b9-48e5-9886-49ad5f49611b","Type":"ContainerStarted","Data":"a9f8433f6e165eb03b36a1d73980c1fe8b7c753cf409e18f54c308af003b7f43"}
Jul 02 13:17:24.389167 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:24.389133    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b67948f45-sgmdm" event={"ID":"0189327b-65b9-48e5-9886-49ad5f49611b","Type":"ContainerStarted","Data":"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"}
Jul 02 13:17:24.409494 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:24.409448    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b67948f45-sgmdm" podStartSLOduration=2.409432924 podStartE2EDuration="2.409432924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:24.40836896 +0000 UTC m=+156.263761881" watchObservedRunningTime="2026-07-02 13:17:24.409432924 +0000 UTC m=+156.264825806"
Jul 02 13:17:33.219044 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:33.219015    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:33.219716 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:33.219067    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:33.224145 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:33.224124    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:33.428593 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:33.428569    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:17:33.504269 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:33.504198    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:17:58.529582 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.529506    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6bc765bc4-pr8pj" podUID="b54bb418-31a8-4957-aa3d-69f866eae3d9" containerName="console" containerID="cri-o://c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796" gracePeriod=15
Jul 02 13:17:58.768130 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.768108    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:17:58.864016 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.863949    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rb6l\" (UniqueName: \"kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864016 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.863995    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864022    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864046    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864099    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864132    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864181    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert\") pod \"b54bb418-31a8-4957-aa3d-69f866eae3d9\" (UID: \"b54bb418-31a8-4957-aa3d-69f866eae3d9\") "
Jul 02 13:17:58.864627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864568    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config" (OuterVolumeSpecName: "console-config") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:58.864627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864602    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:58.864627 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864599    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:58.864822 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864664    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.864822 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.864708    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:58.866252 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.866228    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:58.866333 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.866260    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l" (OuterVolumeSpecName: "kube-api-access-4rb6l") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "kube-api-access-4rb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:58.866333 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.866293    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b54bb418-31a8-4957-aa3d-69f866eae3d9" (UID: "b54bb418-31a8-4957-aa3d-69f866eae3d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:58.964983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.964956    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rb6l\" (UniqueName: \"kubernetes.io/projected/b54bb418-31a8-4957-aa3d-69f866eae3d9-kube-api-access-4rb6l\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.964983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.964984    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.965100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.964994    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54bb418-31a8-4957-aa3d-69f866eae3d9-console-oauth-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.965100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.965003    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-service-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.965100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.965013    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-trusted-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:58.965100 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:58.965021    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54bb418-31a8-4957-aa3d-69f866eae3d9-oauth-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.526987 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.526953    2552 generic.go:356] "Generic (PLEG): container finished" podID="b54bb418-31a8-4957-aa3d-69f866eae3d9" containerID="c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796" exitCode=0
Jul 02 13:17:59.527133 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.527001    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc765bc4-pr8pj"
Jul 02 13:17:59.527133 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.527043    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc765bc4-pr8pj" event={"ID":"b54bb418-31a8-4957-aa3d-69f866eae3d9","Type":"ContainerDied","Data":"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"}
Jul 02 13:17:59.527133 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.527100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc765bc4-pr8pj" event={"ID":"b54bb418-31a8-4957-aa3d-69f866eae3d9","Type":"ContainerDied","Data":"77b3cda45749beef16f8a7143c844824cb566fb2761ce5ed84cca43536891f2d"}
Jul 02 13:17:59.527133 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.527120    2552 scope.go:117] "RemoveContainer" containerID="c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"
Jul 02 13:17:59.535561 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.535382    2552 scope.go:117] "RemoveContainer" containerID="c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"
Jul 02 13:17:59.535773 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:17:59.535655    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796\": container with ID starting with c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796 not found: ID does not exist" containerID="c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"
Jul 02 13:17:59.535773 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.535678    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796"} err="failed to get container status \"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796\": rpc error: code = NotFound desc = could not find container \"c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796\": container with ID starting with c5ad4ddd794d1cb8faab33ee8af9030518a7eb6841993b1b0348b67fbbb2b796 not found: ID does not exist"
Jul 02 13:17:59.547682 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.547657    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:17:59.553531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.553511    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bc765bc4-pr8pj"]
Jul 02 13:17:59.565797 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:17:59.565763    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b54bb418-31a8-4957-aa3d-69f866eae3d9" path="/var/lib/kubelet/pods/b54bb418-31a8-4957-aa3d-69f866eae3d9/volumes"
Jul 02 13:18:11.994630 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:18:11.994588    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:18:12.010284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:18:12.010253    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:18:12.596939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:18:12.596908    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:19:37.009570 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.009501    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-b5q4q"]
Jul 02 13:19:37.010015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.009851    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54bb418-31a8-4957-aa3d-69f866eae3d9" containerName="console"
Jul 02 13:19:37.010015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.009864    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54bb418-31a8-4957-aa3d-69f866eae3d9" containerName="console"
Jul 02 13:19:37.010015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.009931    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b54bb418-31a8-4957-aa3d-69f866eae3d9" containerName="console"
Jul 02 13:19:37.012779 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.012763    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.015305 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.015285    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 13:19:37.015773 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.015752    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-s7dc8\""
Jul 02 13:19:37.016320 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.016304    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 13:19:37.016552 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.016541    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 13:19:37.016612 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.016598    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 13:19:37.026775 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.026756    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-b5q4q"]
Jul 02 13:19:37.059479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.059456    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.059570 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.059486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztxm\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-kube-api-access-qztxm\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.160255 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.160230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztxm\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-kube-api-access-qztxm\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.160368 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.160320    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.160425 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:19:37.160412    2552 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:37.160461 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:19:37.160435    2552 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-b5q4q: secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:37.160494 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:19:37.160486    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates podName:6c0e2457-1890-47df-a32b-43d7acea398f nodeName:}" failed. No retries permitted until 2026-07-02 13:19:37.660470216 +0000 UTC m=+289.515863087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates") pod "keda-admission-5fbd5c4644-b5q4q" (UID: "6c0e2457-1890-47df-a32b-43d7acea398f") : secret "keda-admission-webhooks-certs" not found
Jul 02 13:19:37.170906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.170886    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztxm\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-kube-api-access-qztxm\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.663680 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.663646    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.665921 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.665899    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c0e2457-1890-47df-a32b-43d7acea398f-certificates\") pod \"keda-admission-5fbd5c4644-b5q4q\" (UID: \"6c0e2457-1890-47df-a32b-43d7acea398f\") " pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:37.923237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:37.923159    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:38.062187 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:38.062163    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-b5q4q"]
Jul 02 13:19:38.064686 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:19:38.064653    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0e2457_1890_47df_a32b_43d7acea398f.slice/crio-e1414b53f1dd21e7ecdb28ddfc8bf6782bea7094b9f368eec0cebda3f1fad332 WatchSource:0}: Error finding container e1414b53f1dd21e7ecdb28ddfc8bf6782bea7094b9f368eec0cebda3f1fad332: Status 404 returned error can't find the container with id e1414b53f1dd21e7ecdb28ddfc8bf6782bea7094b9f368eec0cebda3f1fad332
Jul 02 13:19:38.910590 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:38.910552    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q" event={"ID":"6c0e2457-1890-47df-a32b-43d7acea398f","Type":"ContainerStarted","Data":"e1414b53f1dd21e7ecdb28ddfc8bf6782bea7094b9f368eec0cebda3f1fad332"}
Jul 02 13:19:41.924476 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:41.924438    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q" event={"ID":"6c0e2457-1890-47df-a32b-43d7acea398f","Type":"ContainerStarted","Data":"a60d203ea7a32f19c82aa01b50aca7823b05f932f4205b621bd9873dc57a11b1"}
Jul 02 13:19:41.924866 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:41.924557    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:19:41.941516 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:41.941471    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q" podStartSLOduration=3.114786628 podStartE2EDuration="5.941454258s" totalImagesPullingTime="2.82666763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:41.941263912 +0000 UTC m=+293.796656816" watchObservedRunningTime="2026-07-02 13:19:41.941454258 +0000 UTC m=+293.796847142"
Jul 02 13:19:48.542979 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:19:48.542949    2552 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:20:02.930452 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:20:02.930419    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-b5q4q"
Jul 02 13:21:07.841411 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.841382    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-869d497466-z5474"]
Jul 02 13:21:07.844724 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.844706    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.861528 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.861507    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869d497466-z5474"]
Jul 02 13:21:07.950321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-oauth-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950447 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950447 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-oauth-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950447 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950372    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-service-ca\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950447 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950432    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950583 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950460    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-trusted-ca-bundle\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:07.950583 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:07.950490    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcn7\" (UniqueName: \"kubernetes.io/projected/bec8005f-d24d-40ba-b0e6-41aa5f671828-kube-api-access-lqcn7\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.050841 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.050977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-oauth-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.050977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050882    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-service-ca\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.050977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050916    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.050977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050943    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-trusted-ca-bundle\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051198 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.050981    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcn7\" (UniqueName: \"kubernetes.io/projected/bec8005f-d24d-40ba-b0e6-41aa5f671828-kube-api-access-lqcn7\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051198 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.051036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-oauth-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.051620    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-service-ca\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051761 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.051669    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-oauth-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051761 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.051686    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.051979 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.051959    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec8005f-d24d-40ba-b0e6-41aa5f671828-trusted-ca-bundle\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.053250 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.053223    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-serving-cert\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.053352 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.053336    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bec8005f-d24d-40ba-b0e6-41aa5f671828-console-oauth-config\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.060270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.060251    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcn7\" (UniqueName: \"kubernetes.io/projected/bec8005f-d24d-40ba-b0e6-41aa5f671828-kube-api-access-lqcn7\") pod \"console-869d497466-z5474\" (UID: \"bec8005f-d24d-40ba-b0e6-41aa5f671828\") " pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.153934 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.153850    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:08.293463 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:08.293435    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869d497466-z5474"]
Jul 02 13:21:08.296312 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:21:08.296283    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec8005f_d24d_40ba_b0e6_41aa5f671828.slice/crio-454d7fe112f22525ed9cb21f0474044756b0e961fdcfb880026b3955b286e3ea WatchSource:0}: Error finding container 454d7fe112f22525ed9cb21f0474044756b0e961fdcfb880026b3955b286e3ea: Status 404 returned error can't find the container with id 454d7fe112f22525ed9cb21f0474044756b0e961fdcfb880026b3955b286e3ea
Jul 02 13:21:09.268801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:09.268769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d497466-z5474" event={"ID":"bec8005f-d24d-40ba-b0e6-41aa5f671828","Type":"ContainerStarted","Data":"caf60c4c723202943f71f05dc68cbe70c7489878d3d7d89f267682c541a22e05"}
Jul 02 13:21:09.269186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:09.268815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d497466-z5474" event={"ID":"bec8005f-d24d-40ba-b0e6-41aa5f671828","Type":"ContainerStarted","Data":"454d7fe112f22525ed9cb21f0474044756b0e961fdcfb880026b3955b286e3ea"}
Jul 02 13:21:18.154857 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.154824    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:18.155373 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.154896    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:18.159644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.159622    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:18.186537 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.186487    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-869d497466-z5474" podStartSLOduration=11.186476204 podStartE2EDuration="11.186476204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:09.291098147 +0000 UTC m=+381.146491033" watchObservedRunningTime="2026-07-02 13:21:18.186476204 +0000 UTC m=+390.041869160"
Jul 02 13:21:18.307778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.307757    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-869d497466-z5474"
Jul 02 13:21:18.377405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:18.377381    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:21:22.000798 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.000768    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-v2s6v"]
Jul 02 13:21:22.004377 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.004355    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.007146 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.007121    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:21:22.008182 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.008157    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-xkdlx\""
Jul 02 13:21:22.008182 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.008176    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:21:22.008342 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.008297    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 13:21:22.013735 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.013715    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-v2s6v"]
Jul 02 13:21:22.053927 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.053904    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db2df7fa-b894-45f2-8d73-b4a88d2d789b-cert\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.054026 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.053958    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kngz\" (UniqueName: \"kubernetes.io/projected/db2df7fa-b894-45f2-8d73-b4a88d2d789b-kube-api-access-8kngz\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.155137 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.155113    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db2df7fa-b894-45f2-8d73-b4a88d2d789b-cert\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.155220 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.155157    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kngz\" (UniqueName: \"kubernetes.io/projected/db2df7fa-b894-45f2-8d73-b4a88d2d789b-kube-api-access-8kngz\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.157382 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.157366    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db2df7fa-b894-45f2-8d73-b4a88d2d789b-cert\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.163666 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.163647    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kngz\" (UniqueName: \"kubernetes.io/projected/db2df7fa-b894-45f2-8d73-b4a88d2d789b-kube-api-access-8kngz\") pod \"kserve-controller-manager-694f5b7594-v2s6v\" (UID: \"db2df7fa-b894-45f2-8d73-b4a88d2d789b\") " pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.315193 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.315123    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:22.444270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.444241    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-v2s6v"]
Jul 02 13:21:22.448052 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:21:22.448027    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2df7fa_b894_45f2_8d73_b4a88d2d789b.slice/crio-d32bc88caebb14017564b3689dc6d000448597ff9d5cc71c5a2cf639ef289f64 WatchSource:0}: Error finding container d32bc88caebb14017564b3689dc6d000448597ff9d5cc71c5a2cf639ef289f64: Status 404 returned error can't find the container with id d32bc88caebb14017564b3689dc6d000448597ff9d5cc71c5a2cf639ef289f64
Jul 02 13:21:22.449379 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:22.449363    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:21:23.326738 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:23.326701    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v" event={"ID":"db2df7fa-b894-45f2-8d73-b4a88d2d789b","Type":"ContainerStarted","Data":"d32bc88caebb14017564b3689dc6d000448597ff9d5cc71c5a2cf639ef289f64"}
Jul 02 13:21:25.336051 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:25.336021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v" event={"ID":"db2df7fa-b894-45f2-8d73-b4a88d2d789b","Type":"ContainerStarted","Data":"6ca2d846d43eeb697e8892a13981683c3f8ddb5122a8081f139a566a066b7b0f"}
Jul 02 13:21:25.336450 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:25.336151    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:25.354208 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:25.354164    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v" podStartSLOduration=1.865089528 podStartE2EDuration="4.354150392s" totalImagesPullingTime="2.489060864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:25.352822088 +0000 UTC m=+397.208214972" watchObservedRunningTime="2026-07-02 13:21:25.354150392 +0000 UTC m=+397.209543274"
Jul 02 13:21:43.397521 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.397463    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-b67948f45-sgmdm" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" containerName="console" containerID="cri-o://25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce" gracePeriod=15
Jul 02 13:21:43.426178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.426150    2552 patch_prober.go:28] interesting pod/console-b67948f45-sgmdm container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" start-of-body=
Jul 02 13:21:43.426269 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.426201    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-b67948f45-sgmdm" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" containerName="console" probeResult="failure" output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused"
Jul 02 13:21:43.636548 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.636522    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:21:43.718351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718365    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718386    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718423    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8jtr\" (UniqueName: \"kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718440    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718651 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718483    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718651 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718508    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca\") pod \"0189327b-65b9-48e5-9886-49ad5f49611b\" (UID: \"0189327b-65b9-48e5-9886-49ad5f49611b\") "
Jul 02 13:21:43.718915 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718846    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config" (OuterVolumeSpecName: "console-config") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:43.718988 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.718855    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:43.719036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.719018    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca" (OuterVolumeSpecName: "service-ca") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:43.719170 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.719155    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:43.720615 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.720592    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:43.720697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.720620    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:43.720697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.720633    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr" (OuterVolumeSpecName: "kube-api-access-q8jtr") pod "0189327b-65b9-48e5-9886-49ad5f49611b" (UID: "0189327b-65b9-48e5-9886-49ad5f49611b"). InnerVolumeSpecName "kube-api-access-q8jtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:21:43.819865 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819843    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-oauth-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.819865 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819864    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-service-ca\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.820013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819889    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-trusted-ca-bundle\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.820013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819899    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0189327b-65b9-48e5-9886-49ad5f49611b-console-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.820013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819907    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-oauth-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.820013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819915    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jtr\" (UniqueName: \"kubernetes.io/projected/0189327b-65b9-48e5-9886-49ad5f49611b-kube-api-access-q8jtr\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:43.820013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:43.819924    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0189327b-65b9-48e5-9886-49ad5f49611b-console-serving-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:21:44.413513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.413478    2552 generic.go:356] "Generic (PLEG): container finished" podID="0189327b-65b9-48e5-9886-49ad5f49611b" containerID="25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce" exitCode=0
Jul 02 13:21:44.414022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.413529    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b67948f45-sgmdm"
Jul 02 13:21:44.414022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.413560    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b67948f45-sgmdm" event={"ID":"0189327b-65b9-48e5-9886-49ad5f49611b","Type":"ContainerDied","Data":"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"}
Jul 02 13:21:44.414022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.413608    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b67948f45-sgmdm" event={"ID":"0189327b-65b9-48e5-9886-49ad5f49611b","Type":"ContainerDied","Data":"a9f8433f6e165eb03b36a1d73980c1fe8b7c753cf409e18f54c308af003b7f43"}
Jul 02 13:21:44.414022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.413631    2552 scope.go:117] "RemoveContainer" containerID="25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"
Jul 02 13:21:44.422209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.422194    2552 scope.go:117] "RemoveContainer" containerID="25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"
Jul 02 13:21:44.422434 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:21:44.422419    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce\": container with ID starting with 25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce not found: ID does not exist" containerID="25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"
Jul 02 13:21:44.422491 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.422441    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce"} err="failed to get container status \"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce\": rpc error: code = NotFound desc = could not find container \"25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce\": container with ID starting with 25d4368b888d5af225dc39e37ae25b1c172905e3429c34923872551e902313ce not found: ID does not exist"
Jul 02 13:21:44.436312 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.436286    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:21:44.440935 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:44.440914    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b67948f45-sgmdm"]
Jul 02 13:21:45.571622 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:45.571588    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" path="/var/lib/kubelet/pods/0189327b-65b9-48e5-9886-49ad5f49611b/volumes"
Jul 02 13:21:56.344846 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:56.344812    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-v2s6v"
Jul 02 13:21:57.414146 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.414114    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-2scph"]
Jul 02 13:21:57.414489 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.414479    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" containerName="console"
Jul 02 13:21:57.414529 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.414492    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" containerName="console"
Jul 02 13:21:57.414569 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.414559    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0189327b-65b9-48e5-9886-49ad5f49611b" containerName="console"
Jul 02 13:21:57.418983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.418965    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.421458 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.421436    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 13:21:57.421458 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.421444    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lcrkr\""
Jul 02 13:21:57.428176 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.428155    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-2scph"]
Jul 02 13:21:57.522252 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.522227    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-cert\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.522367 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.522272    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dv4\" (UniqueName: \"kubernetes.io/projected/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-kube-api-access-c5dv4\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.622631 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.622607    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-cert\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.622740 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.622649    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dv4\" (UniqueName: \"kubernetes.io/projected/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-kube-api-access-c5dv4\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.624744 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.624719    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-cert\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.630313 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.630292    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dv4\" (UniqueName: \"kubernetes.io/projected/a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541-kube-api-access-c5dv4\") pod \"odh-model-controller-84b54b7f97-2scph\" (UID: \"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541\") " pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.729089 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.729068    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:21:57.853695 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:57.853673    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-2scph"]
Jul 02 13:21:57.856505 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:21:57.856474    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda286c0f3_5d8b_4cd0_a7ec_6b3a8b511541.slice/crio-61ad5ed79a0e6f7b9e811b699f0795d3d7702e31562c1937a66439a965be8ff0 WatchSource:0}: Error finding container 61ad5ed79a0e6f7b9e811b699f0795d3d7702e31562c1937a66439a965be8ff0: Status 404 returned error can't find the container with id 61ad5ed79a0e6f7b9e811b699f0795d3d7702e31562c1937a66439a965be8ff0
Jul 02 13:21:58.483319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:21:58.483279    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-2scph" event={"ID":"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541","Type":"ContainerStarted","Data":"61ad5ed79a0e6f7b9e811b699f0795d3d7702e31562c1937a66439a965be8ff0"}
Jul 02 13:22:01.500178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:01.500141    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-2scph" event={"ID":"a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541","Type":"ContainerStarted","Data":"d74af064e7966e78f6c654c4e151319658d0bc1eb8d61ac299aaf51a5099c495"}
Jul 02 13:22:01.500584 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:01.500216    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:22:01.521614 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:01.521568    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-2scph" podStartSLOduration=1.741645139 podStartE2EDuration="4.521546144s" totalImagesPullingTime="2.779901005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:01.520250798 +0000 UTC m=+433.375643822" watchObservedRunningTime="2026-07-02 13:22:01.521546144 +0000 UTC m=+433.376939027"
Jul 02 13:22:12.506301 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:12.506224    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-2scph"
Jul 02 13:22:13.460662 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.460631    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pt9rk"]
Jul 02 13:22:13.463867 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.463849    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.466721 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.466703    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ls2bv\""
Jul 02 13:22:13.466778 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.466751    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 13:22:13.473890 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.473856    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pt9rk"]
Jul 02 13:22:13.545695 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.545677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.546005 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.545719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zxb\" (UniqueName: \"kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.646382 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.646358    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.646482 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.646418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zxb\" (UniqueName: \"kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.646737 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.646720    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.656140 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.656115    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zxb\" (UniqueName: \"kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb\") pod \"s3-init-pt9rk\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") " pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.779206 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.779187    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9rk"
Jul 02 13:22:13.904834 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:13.904808    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pt9rk"]
Jul 02 13:22:13.907754 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:22:13.907728    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddedef70c_57a2_450c_8bc0_45c5e2443ce2.slice/crio-f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c WatchSource:0}: Error finding container f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c: Status 404 returned error can't find the container with id f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c
Jul 02 13:22:14.552453 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:14.552418    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerStarted","Data":"f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c"}
Jul 02 13:22:19.574117 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:19.574083    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerStarted","Data":"7031d33042f9d3103d9b7c478c6419a4c73ad0ba4acd5fbc7386f7b88c043072"}
Jul 02 13:22:20.579898 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:20.579846    2552 generic.go:356] "Generic (PLEG): container finished" podID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerID="7031d33042f9d3103d9b7c478c6419a4c73ad0ba4acd5fbc7386f7b88c043072" exitCode=0
Jul 02 13:22:20.580228 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:20.579906    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerDied","Data":"7031d33042f9d3103d9b7c478c6419a4c73ad0ba4acd5fbc7386f7b88c043072"}
Jul 02 13:22:26.609483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:26.609431    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerStarted","Data":"49d3c823b5f050a9e92afdecea1379174c4ef32f7eb9ef4bc47018256a145d9e"}
Jul 02 13:22:26.628913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:22:26.628825    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pt9rk" podStartSLOduration=2.026212536 podStartE2EDuration="13.628811367s" totalImagesPullingTime="10.263272633s" totalInitContainerRuntime="1.339326198s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:26.627849383 +0000 UTC m=+458.483242265" watchObservedRunningTime="2026-07-02 13:22:26.628811367 +0000 UTC m=+458.484204307"
Jul 02 13:23:30.865630 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:30.865597    2552 generic.go:356] "Generic (PLEG): container finished" podID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerID="49d3c823b5f050a9e92afdecea1379174c4ef32f7eb9ef4bc47018256a145d9e" exitCode=0
Jul 02 13:23:30.866031 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:30.865672    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerDied","Data":"49d3c823b5f050a9e92afdecea1379174c4ef32f7eb9ef4bc47018256a145d9e"}
Jul 02 13:23:31.995334 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:31.995309    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9rk"
Jul 02 13:23:32.093560 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.093528    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache\") pod \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") "
Jul 02 13:23:32.093718 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.093625    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2zxb\" (UniqueName: \"kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb\") pod \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\" (UID: \"dedef70c-57a2-450c-8bc0-45c5e2443ce2\") "
Jul 02 13:23:32.095664 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.095634    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb" (OuterVolumeSpecName: "kube-api-access-l2zxb") pod "dedef70c-57a2-450c-8bc0-45c5e2443ce2" (UID: "dedef70c-57a2-450c-8bc0-45c5e2443ce2"). InnerVolumeSpecName "kube-api-access-l2zxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:32.111728 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.111702    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache" (OuterVolumeSpecName: "model-cache") pod "dedef70c-57a2-450c-8bc0-45c5e2443ce2" (UID: "dedef70c-57a2-450c-8bc0-45c5e2443ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:32.194709 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.194653    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedef70c-57a2-450c-8bc0-45c5e2443ce2-model-cache\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:32.194709 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.194674    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2zxb\" (UniqueName: \"kubernetes.io/projected/dedef70c-57a2-450c-8bc0-45c5e2443ce2-kube-api-access-l2zxb\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:32.874816 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.874788    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pt9rk"
Jul 02 13:23:32.875000 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.874788    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pt9rk" event={"ID":"dedef70c-57a2-450c-8bc0-45c5e2443ce2","Type":"ContainerDied","Data":"f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c"}
Jul 02 13:23:32.875000 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:32.874910    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0611181aa0fd7d0ffad57ded90d405ed3e039b0b652fb116851c70b6b3eeb7c"
Jul 02 13:23:34.352588 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.352554    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:34.352993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.352960    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerName="download-hf-model"
Jul 02 13:23:34.352993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.352972    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerName="download-hf-model"
Jul 02 13:23:34.352993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.352982    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerName="s3-init"
Jul 02 13:23:34.352993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.352988    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerName="s3-init"
Jul 02 13:23:34.353128 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.353037    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedef70c-57a2-450c-8bc0-45c5e2443ce2" containerName="s3-init"
Jul 02 13:23:34.356105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.356086    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.358617 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.358597    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 13:23:34.359035 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.359020    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ls2bv\""
Jul 02 13:23:34.371168 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.371145    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:34.513973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.513936    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.514142 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.513984    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxw58\" (UniqueName: \"kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.614632 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.614548    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.614632 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.614593    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxw58\" (UniqueName: \"kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.614979 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.614959    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.623809 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.623784    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxw58\" (UniqueName: \"kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58\") pod \"seaweedfs-tls-custom-6644bf5d6c-q6rcv\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.664861 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.664837    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:34.792868 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.792833    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:34.795826 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:23:34.795797    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4cdd02_06c1_40f4_bd29_cbbc4936d3f9.slice/crio-614b934a47e56adb9566ead3f7dfc1899a71e2396e20506f08bb4af8a3387fa4 WatchSource:0}: Error finding container 614b934a47e56adb9566ead3f7dfc1899a71e2396e20506f08bb4af8a3387fa4: Status 404 returned error can't find the container with id 614b934a47e56adb9566ead3f7dfc1899a71e2396e20506f08bb4af8a3387fa4
Jul 02 13:23:34.884923 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:34.884845    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" event={"ID":"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9","Type":"ContainerStarted","Data":"614b934a47e56adb9566ead3f7dfc1899a71e2396e20506f08bb4af8a3387fa4"}
Jul 02 13:23:38.906222 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:38.906186    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" event={"ID":"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9","Type":"ContainerStarted","Data":"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"}
Jul 02 13:23:38.925021 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:38.924964    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" podStartSLOduration=1.671849163 podStartE2EDuration="4.924941803s" totalImagesPullingTime="3.25309264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:38.924399966 +0000 UTC m=+530.779792849" watchObservedRunningTime="2026-07-02 13:23:38.924941803 +0000 UTC m=+530.780334672"
Jul 02 13:23:40.513526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:40.513439    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:40.914524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:40.914427    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" podUID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" containerName="seaweedfs-tls-custom" containerID="cri-o://cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d" gracePeriod=30
Jul 02 13:23:42.158618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.158595    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:42.293718 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.293690    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxw58\" (UniqueName: \"kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58\") pod \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") "
Jul 02 13:23:42.293908 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.293765    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data\") pod \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\" (UID: \"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9\") "
Jul 02 13:23:42.295049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.295025    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data" (OuterVolumeSpecName: "data") pod "fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" (UID: "fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:42.295700 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.295673    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58" (OuterVolumeSpecName: "kube-api-access-hxw58") pod "fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" (UID: "fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9"). InnerVolumeSpecName "kube-api-access-hxw58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:42.395142 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.395113    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxw58\" (UniqueName: \"kubernetes.io/projected/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-kube-api-access-hxw58\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:42.395142 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.395140    2552 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9-data\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:42.923752 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.923718    2552 generic.go:356] "Generic (PLEG): container finished" podID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" containerID="cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d" exitCode=0
Jul 02 13:23:42.923929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.923758    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"
Jul 02 13:23:42.923929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.923798    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" event={"ID":"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9","Type":"ContainerDied","Data":"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"}
Jul 02 13:23:42.923929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.923846    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv" event={"ID":"fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9","Type":"ContainerDied","Data":"614b934a47e56adb9566ead3f7dfc1899a71e2396e20506f08bb4af8a3387fa4"}
Jul 02 13:23:42.923929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.923899    2552 scope.go:117] "RemoveContainer" containerID="cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"
Jul 02 13:23:42.933693 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.933677    2552 scope.go:117] "RemoveContainer" containerID="cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"
Jul 02 13:23:42.933929 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:23:42.933904    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d\": container with ID starting with cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d not found: ID does not exist" containerID="cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"
Jul 02 13:23:42.933984 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.933938    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d"} err="failed to get container status \"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d\": rpc error: code = NotFound desc = could not find container \"cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d\": container with ID starting with cd8f0da595ab82f03fbaaa0ee3f278726afe0c3f06b45d589bb62e832778e21d not found: ID does not exist"
Jul 02 13:23:42.945938 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.945917    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:42.950434 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:42.950410    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-q6rcv"]
Jul 02 13:23:43.565865 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:43.565828    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" path="/var/lib/kubelet/pods/fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9/volumes"
Jul 02 13:23:44.770470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.770440    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6hq7f"]
Jul 02 13:23:44.770836 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.770798    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" containerName="seaweedfs-tls-custom"
Jul 02 13:23:44.770836 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.770809    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" containerName="seaweedfs-tls-custom"
Jul 02 13:23:44.770954 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.770858    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4cdd02-06c1-40f4-bd29-cbbc4936d3f9" containerName="seaweedfs-tls-custom"
Jul 02 13:23:44.775172 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.775154    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.777569 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.777547    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 13:23:44.777569 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.777567    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ls2bv\""
Jul 02 13:23:44.782400 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.782233    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6hq7f"]
Jul 02 13:23:44.811048 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.811024    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cw8\" (UniqueName: \"kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.811139 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.811086    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.911843 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.911818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cw8\" (UniqueName: \"kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.911976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.911867    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.912204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.912189    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:44.920216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:44.920195    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cw8\" (UniqueName: \"kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8\") pod \"s3-tls-init-custom-6hq7f\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") " pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:45.096602 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:45.096534    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:45.225281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:45.225260    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6hq7f"]
Jul 02 13:23:45.227832 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:23:45.227804    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod264811d2_d4ff_4f91_af26_b0aea8898068.slice/crio-cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924 WatchSource:0}: Error finding container cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924: Status 404 returned error can't find the container with id cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924
Jul 02 13:23:45.938189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:45.938156    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerStarted","Data":"c3f5363e3fa2f0ce012188d6601d7b291ac46fd47d1b30b33aa6cdb9e514882e"}
Jul 02 13:23:45.938189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:45.938198    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerStarted","Data":"cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924"}
Jul 02 13:23:47.948462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:47.948434    2552 generic.go:356] "Generic (PLEG): container finished" podID="264811d2-d4ff-4f91-af26-b0aea8898068" containerID="c3f5363e3fa2f0ce012188d6601d7b291ac46fd47d1b30b33aa6cdb9e514882e" exitCode=0
Jul 02 13:23:47.948757 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:47.948507    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerDied","Data":"c3f5363e3fa2f0ce012188d6601d7b291ac46fd47d1b30b33aa6cdb9e514882e"}
Jul 02 13:23:48.954307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:48.954230    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerStarted","Data":"b031ac2e7228170667e7647c99b9b9acdb11eb3babe574bf7b73b5102c387241"}
Jul 02 13:23:48.974495 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:48.974450    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6hq7f" podStartSLOduration=2.416901921 podStartE2EDuration="4.974436322s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.557534401s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:48.972294655 +0000 UTC m=+540.827687539" watchObservedRunningTime="2026-07-02 13:23:48.974436322 +0000 UTC m=+540.829829228"
Jul 02 13:23:53.977636 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:53.977607    2552 generic.go:356] "Generic (PLEG): container finished" podID="264811d2-d4ff-4f91-af26-b0aea8898068" containerID="b031ac2e7228170667e7647c99b9b9acdb11eb3babe574bf7b73b5102c387241" exitCode=0
Jul 02 13:23:53.978028 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:53.977645    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerDied","Data":"b031ac2e7228170667e7647c99b9b9acdb11eb3babe574bf7b73b5102c387241"}
Jul 02 13:23:55.123594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.123571    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:55.199560 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.199536    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache\") pod \"264811d2-d4ff-4f91-af26-b0aea8898068\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") "
Jul 02 13:23:55.199679 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.199601    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78cw8\" (UniqueName: \"kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8\") pod \"264811d2-d4ff-4f91-af26-b0aea8898068\" (UID: \"264811d2-d4ff-4f91-af26-b0aea8898068\") "
Jul 02 13:23:55.201518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.201488    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8" (OuterVolumeSpecName: "kube-api-access-78cw8") pod "264811d2-d4ff-4f91-af26-b0aea8898068" (UID: "264811d2-d4ff-4f91-af26-b0aea8898068"). InnerVolumeSpecName "kube-api-access-78cw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:55.221572 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.221543    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache" (OuterVolumeSpecName: "model-cache") pod "264811d2-d4ff-4f91-af26-b0aea8898068" (UID: "264811d2-d4ff-4f91-af26-b0aea8898068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:55.300738 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.300687    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cw8\" (UniqueName: \"kubernetes.io/projected/264811d2-d4ff-4f91-af26-b0aea8898068-kube-api-access-78cw8\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:55.300738 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.300707    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264811d2-d4ff-4f91-af26-b0aea8898068-model-cache\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:23:55.988486 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.988456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6hq7f" event={"ID":"264811d2-d4ff-4f91-af26-b0aea8898068","Type":"ContainerDied","Data":"cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924"}
Jul 02 13:23:55.988486 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.988481    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6hq7f"
Jul 02 13:23:55.988655 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:55.988492    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5377a9ff06d90bb83dc09608e610a7ae71a1d357772a49af6b0f743247a924"
Jul 02 13:23:59.034535 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034502    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-82h99"]
Jul 02 13:23:59.034913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034813    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264811d2-d4ff-4f91-af26-b0aea8898068" containerName="download-hf-model"
Jul 02 13:23:59.034913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034824    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="264811d2-d4ff-4f91-af26-b0aea8898068" containerName="download-hf-model"
Jul 02 13:23:59.034913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034836    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264811d2-d4ff-4f91-af26-b0aea8898068" containerName="s3-tls-init-custom"
Jul 02 13:23:59.034913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034841    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="264811d2-d4ff-4f91-af26-b0aea8898068" containerName="s3-tls-init-custom"
Jul 02 13:23:59.035059 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.034930    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="264811d2-d4ff-4f91-af26-b0aea8898068" containerName="s3-tls-init-custom"
Jul 02 13:23:59.040441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.040424    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.043067 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.043047    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ls2bv\""
Jul 02 13:23:59.043170 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.043050    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 02 13:23:59.046615 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.046588    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-82h99"]
Jul 02 13:23:59.130487 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.130461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.130632 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.130521    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6bdh\" (UniqueName: \"kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.231832 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.231810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6bdh\" (UniqueName: \"kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.231959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.231849    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.232167 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.232152    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.240075 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.240046    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6bdh\" (UniqueName: \"kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh\") pod \"s3-tls-init-serving-82h99\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") " pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.362049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.361996    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:23:59.489693 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:23:59.489671    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-82h99"]
Jul 02 13:23:59.492135 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:23:59.492110    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4af3453_caf5_4441_aeb2_cd3103bf8a75.slice/crio-b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11 WatchSource:0}: Error finding container b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11: Status 404 returned error can't find the container with id b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11
Jul 02 13:24:00.008444 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:00.008410    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerStarted","Data":"a4c8dae8cef3273523e5ce512749aecd22ba9be5a12a43c757a5a095c1f435af"}
Jul 02 13:24:00.008595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:00.008454    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerStarted","Data":"b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11"}
Jul 02 13:24:01.014278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:01.014251    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerID="a4c8dae8cef3273523e5ce512749aecd22ba9be5a12a43c757a5a095c1f435af" exitCode=0
Jul 02 13:24:01.014597 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:01.014332    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerDied","Data":"a4c8dae8cef3273523e5ce512749aecd22ba9be5a12a43c757a5a095c1f435af"}
Jul 02 13:24:02.020257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:02.020225    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerStarted","Data":"2237f19ee399e538ecb65e2b7a49d5edf114a225588205ed3ee45e9ab1eac9ff"}
Jul 02 13:24:02.040956 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:02.040912    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-82h99" podStartSLOduration=1.611094713 podStartE2EDuration="3.040897094s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.429802381s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:02.039589821 +0000 UTC m=+553.894982705" watchObservedRunningTime="2026-07-02 13:24:02.040897094 +0000 UTC m=+553.896289977"
Jul 02 13:24:07.043257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:07.043229    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerID="2237f19ee399e538ecb65e2b7a49d5edf114a225588205ed3ee45e9ab1eac9ff" exitCode=0
Jul 02 13:24:07.043569 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:07.043300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerDied","Data":"2237f19ee399e538ecb65e2b7a49d5edf114a225588205ed3ee45e9ab1eac9ff"}
Jul 02 13:24:08.210483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.210461    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:24:08.311189 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.311124    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache\") pod \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") "
Jul 02 13:24:08.311313 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.311221    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6bdh\" (UniqueName: \"kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh\") pod \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\" (UID: \"c4af3453-caf5-4441-aeb2-cd3103bf8a75\") "
Jul 02 13:24:08.313163 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.313139    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh" (OuterVolumeSpecName: "kube-api-access-s6bdh") pod "c4af3453-caf5-4441-aeb2-cd3103bf8a75" (UID: "c4af3453-caf5-4441-aeb2-cd3103bf8a75"). InnerVolumeSpecName "kube-api-access-s6bdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:24:08.337460 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.337431    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4af3453-caf5-4441-aeb2-cd3103bf8a75" (UID: "c4af3453-caf5-4441-aeb2-cd3103bf8a75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:24:08.411967 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.411939    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6bdh\" (UniqueName: \"kubernetes.io/projected/c4af3453-caf5-4441-aeb2-cd3103bf8a75-kube-api-access-s6bdh\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:24:08.411967 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:08.411964    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af3453-caf5-4441-aeb2-cd3103bf8a75-model-cache\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:24:09.053785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:09.053753    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-82h99" event={"ID":"c4af3453-caf5-4441-aeb2-cd3103bf8a75","Type":"ContainerDied","Data":"b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11"}
Jul 02 13:24:09.053785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:09.053779    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-82h99"
Jul 02 13:24:09.053785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:09.053792    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7325d95d2f20bd48c97a7b564a892dab075379aafe7a4ad7b3d39a4d636fc11"
Jul 02 13:24:21.548909 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.548856    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:24:21.549267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.549226    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerName="download-hf-model"
Jul 02 13:24:21.549267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.549238    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerName="download-hf-model"
Jul 02 13:24:21.549267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.549251    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerName="s3-tls-init-serving"
Jul 02 13:24:21.549267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.549256    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerName="s3-tls-init-serving"
Jul 02 13:24:21.549403 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.549361    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4af3453-caf5-4441-aeb2-cd3103bf8a75" containerName="s3-tls-init-serving"
Jul 02 13:24:21.551629 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.551612    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.555443 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.555422    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:24:21.555650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.555637    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:24:21.555857 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.555840    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:24:21.556002 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.555894    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:24:21.556356 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.556319    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 02 13:24:21.556514 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.556499    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 02 13:24:21.556564 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.556529    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f7h2p\""
Jul 02 13:24:21.584236 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.584208    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:24:21.621175 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.621141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.621351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.621186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.621351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.621241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkf6h\" (UniqueName: \"kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.621351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.621277    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.621351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.621325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722153 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkf6h\" (UniqueName: \"kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722275 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722161    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722275 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722188    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722275 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722239    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722275 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722725 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722700    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.722957 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722932    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.723049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.722958    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.724638 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.724621    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.731397 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.731379    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkf6h\" (UniqueName: \"kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h\") pod \"isvc-sklearn-batcher-predictor-7595457646-qn7xn\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:21.862098 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:21.862010    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:22.001271 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:22.001246    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:24:22.004370 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:24:22.004334    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f172d8_f26c_4d81_87dc_3fd6c92af2fe.slice/crio-e42f4308639460cff86760844904bb4da8354e6202ca4cb8b2481c40504cd5ac WatchSource:0}: Error finding container e42f4308639460cff86760844904bb4da8354e6202ca4cb8b2481c40504cd5ac: Status 404 returned error can't find the container with id e42f4308639460cff86760844904bb4da8354e6202ca4cb8b2481c40504cd5ac
Jul 02 13:24:22.111102 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:22.111062    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerStarted","Data":"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"}
Jul 02 13:24:22.111239 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:22.111147    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerStarted","Data":"e42f4308639460cff86760844904bb4da8354e6202ca4cb8b2481c40504cd5ac"}
Jul 02 13:24:26.130245 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:26.130208    2552 generic.go:356] "Generic (PLEG): container finished" podID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerID="fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a" exitCode=0
Jul 02 13:24:26.130614 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:26.130284    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerDied","Data":"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"}
Jul 02 13:24:42.222227 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:42.222193    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerStarted","Data":"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"}
Jul 02 13:24:45.246641 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:45.246608    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerStarted","Data":"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"}
Jul 02 13:24:48.263007 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:48.262966    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerStarted","Data":"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"}
Jul 02 13:24:48.263425 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:48.263077    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:48.286906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:48.286832    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podStartSLOduration=2.593158207 podStartE2EDuration="27.286815079s" totalImagesPullingTime="21.2742788s" totalInitContainerRuntime="3.419378072s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:48.285624703 +0000 UTC m=+600.141017587" watchObservedRunningTime="2026-07-02 13:24:48.286815079 +0000 UTC m=+600.142208165"
Jul 02 13:24:49.267319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:49.267287    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:49.267714 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:49.267384    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:49.268939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:49.268903    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:24:49.269635 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:49.269607    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:24:49.274249 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:49.274231    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:24:50.272087 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:50.272044    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:24:50.272502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:50.272351    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:24:51.275956 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:51.275926    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:24:51.276338 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:24:51.276255    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:01.276666 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:01.276621    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:01.277104 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:01.277076    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:11.276398 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:11.276299    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:11.606714 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:11.276755    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:21.276112 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:21.276072    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:21.276630 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:21.276443    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:31.276112 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:31.276068    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:31.276483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:31.276460    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:41.276207 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:41.276159    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:41.276681 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:41.276593    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:25:51.276789 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:51.276749    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:25:51.277323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:25:51.277299    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:01.277113 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:01.277050    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:01.277562 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:01.277269    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:16.634361 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.634329    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:26:16.634762 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.634571    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" containerID="cri-o://d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03" gracePeriod=30
Jul 02 13:26:16.634762 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.634616    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" containerID="cri-o://f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae" gracePeriod=30
Jul 02 13:26:16.634762 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.634619    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" containerID="cri-o://e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644" gracePeriod=30
Jul 02 13:26:16.662638 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.662613    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:26:16.665342 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.665322    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.669075 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.669054    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 02 13:26:16.669319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.669303    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 02 13:26:16.682729 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.682704    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:26:16.712054 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.712026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.712327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.712063    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.712327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.712124    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.712327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.712148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc279\" (UniqueName: \"kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.712327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.712187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813123 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813096    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc279\" (UniqueName: \"kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813159    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813202    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813226    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813439 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:16.813358    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 02 13:26:16.813439 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:16.813410    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls podName:337c7777-eac4-4665-80f2-f67913d0e4b8 nodeName:}" failed. No retries permitted until 2026-07-02 13:26:17.313389882 +0000 UTC m=+689.168782759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 02 13:26:16.813609 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813584    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813768 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813749    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.813831 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.813782    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:16.821958 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:16.821941    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc279\" (UniqueName: \"kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:17.317466 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.317427    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:17.319983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.319957    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:17.576167 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.576088    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:17.662848 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.662818    2552 generic.go:356] "Generic (PLEG): container finished" podID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerID="f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae" exitCode=2
Jul 02 13:26:17.663229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.662849    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerDied","Data":"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"}
Jul 02 13:26:17.716350 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:17.716323    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:26:17.718076 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:26:17.718048    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337c7777_eac4_4665_80f2_f67913d0e4b8.slice/crio-ba96624925ff829bb3f181355e5080b9aad70233673abdf7fe7e90dfe25e6ad7 WatchSource:0}: Error finding container ba96624925ff829bb3f181355e5080b9aad70233673abdf7fe7e90dfe25e6ad7: Status 404 returned error can't find the container with id ba96624925ff829bb3f181355e5080b9aad70233673abdf7fe7e90dfe25e6ad7
Jul 02 13:26:18.669720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:18.669687    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerStarted","Data":"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"}
Jul 02 13:26:18.670120 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:18.669748    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerStarted","Data":"ba96624925ff829bb3f181355e5080b9aad70233673abdf7fe7e90dfe25e6ad7"}
Jul 02 13:26:19.267696 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:19.267654    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:20.681635 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:20.681606    2552 generic.go:356] "Generic (PLEG): container finished" podID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerID="d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03" exitCode=0
Jul 02 13:26:20.681923 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:20.681679    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerDied","Data":"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"}
Jul 02 13:26:21.276482 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:21.276443    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:26:21.276800 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:21.276768    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:21.687080 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:21.687002    2552 generic.go:356] "Generic (PLEG): container finished" podID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerID="7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250" exitCode=0
Jul 02 13:26:21.687417 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:21.687073    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerDied","Data":"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"}
Jul 02 13:26:22.694801 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.694769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerStarted","Data":"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"}
Jul 02 13:26:22.695186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.694835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerStarted","Data":"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"}
Jul 02 13:26:22.695186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.694860    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerStarted","Data":"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"}
Jul 02 13:26:22.695186 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.695153    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:22.695319 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.695295    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:22.696257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.696235    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:22.718786 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:22.718744    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podStartSLOduration=3.251898364 podStartE2EDuration="6.718729278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.466830914s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:22.718338864 +0000 UTC m=+694.573731770" watchObservedRunningTime="2026-07-02 13:26:22.718729278 +0000 UTC m=+694.574122164"
Jul 02 13:26:23.699036 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:23.698997    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:23.699449 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:23.699042    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:23.699921 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:23.699898    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:24.268321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:24.268281    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:24.704154 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:24.704047    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:24.704571 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:24.704471    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:29.268034 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:29.267989    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:29.268379 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:29.268143    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:29.708883 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:29.708801    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:26:29.709362 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:29.709336    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:29.709635 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:29.709613    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:31.276760 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:31.276712    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:26:31.277197 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:31.277065    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:34.268332 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:34.268295    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:39.267786 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:39.267751    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:39.710184 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:39.710092    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:39.710676 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:39.710650    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:41.276160 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:41.276064    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:26:41.276594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:41.276225    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:41.276594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:41.276432    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:41.276594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:41.276529    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:44.267673 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:44.267633    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:26:46.786362 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.786337    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:46.807715 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.807693    2552 generic.go:356] "Generic (PLEG): container finished" podID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerID="e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644" exitCode=0
Jul 02 13:26:46.807847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.807748    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"
Jul 02 13:26:46.807847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.807766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerDied","Data":"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"}
Jul 02 13:26:46.807847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.807835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn" event={"ID":"52f172d8-f26c-4d81-87dc-3fd6c92af2fe","Type":"ContainerDied","Data":"e42f4308639460cff86760844904bb4da8354e6202ca4cb8b2481c40504cd5ac"}
Jul 02 13:26:46.807993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.807899    2552 scope.go:117] "RemoveContainer" containerID="e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"
Jul 02 13:26:46.819483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.819464    2552 scope.go:117] "RemoveContainer" containerID="f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"
Jul 02 13:26:46.827101 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.827086    2552 scope.go:117] "RemoveContainer" containerID="d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"
Jul 02 13:26:46.835980 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.835963    2552 scope.go:117] "RemoveContainer" containerID="fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"
Jul 02 13:26:46.842987 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.842972    2552 scope.go:117] "RemoveContainer" containerID="e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"
Jul 02 13:26:46.843229 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:46.843208    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644\": container with ID starting with e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644 not found: ID does not exist" containerID="e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"
Jul 02 13:26:46.843290 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843237    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644"} err="failed to get container status \"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644\": rpc error: code = NotFound desc = could not find container \"e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644\": container with ID starting with e0cb49048a51bf56b8149b0e89818f916cb7e4a5e7edd14333a8f123db0f6644 not found: ID does not exist"
Jul 02 13:26:46.843290 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843259    2552 scope.go:117] "RemoveContainer" containerID="f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"
Jul 02 13:26:46.843478 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:46.843462    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae\": container with ID starting with f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae not found: ID does not exist" containerID="f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"
Jul 02 13:26:46.843522 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843483    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae"} err="failed to get container status \"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae\": rpc error: code = NotFound desc = could not find container \"f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae\": container with ID starting with f575bc57876ee73a3c9df687a877aa99ca5fa90e579bba465193403a772c31ae not found: ID does not exist"
Jul 02 13:26:46.843522 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843498    2552 scope.go:117] "RemoveContainer" containerID="d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"
Jul 02 13:26:46.843722 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:46.843704    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03\": container with ID starting with d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03 not found: ID does not exist" containerID="d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"
Jul 02 13:26:46.843793 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843731    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03"} err="failed to get container status \"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03\": rpc error: code = NotFound desc = could not find container \"d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03\": container with ID starting with d70503a0364a4b198fd6cb3bc66b7916597087576bbea7a0921b2fe0fd7f3a03 not found: ID does not exist"
Jul 02 13:26:46.843793 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.843751    2552 scope.go:117] "RemoveContainer" containerID="fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"
Jul 02 13:26:46.844111 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:26:46.844094    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a\": container with ID starting with fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a not found: ID does not exist" containerID="fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"
Jul 02 13:26:46.844178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.844114    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a"} err="failed to get container status \"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a\": rpc error: code = NotFound desc = could not find container \"fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a\": container with ID starting with fcd98bef1b7531544804608d5aabae251dcdf09b34eaf54a8cfc2a5b4742559a not found: ID does not exist"
Jul 02 13:26:46.858430 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858408    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls\") pod \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") "
Jul 02 13:26:46.858515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858445    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config\") pod \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") "
Jul 02 13:26:46.858515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858488    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location\") pod \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") "
Jul 02 13:26:46.858665 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858644    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert\") pod \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") "
Jul 02 13:26:46.858779 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858712    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkf6h\" (UniqueName: \"kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h\") pod \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\" (UID: \"52f172d8-f26c-4d81-87dc-3fd6c92af2fe\") "
Jul 02 13:26:46.858779 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858756    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52f172d8-f26c-4d81-87dc-3fd6c92af2fe" (UID: "52f172d8-f26c-4d81-87dc-3fd6c92af2fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:26:46.858934 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858822    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52f172d8-f26c-4d81-87dc-3fd6c92af2fe" (UID: "52f172d8-f26c-4d81-87dc-3fd6c92af2fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:46.858998 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.858965    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52f172d8-f26c-4d81-87dc-3fd6c92af2fe" (UID: "52f172d8-f26c-4d81-87dc-3fd6c92af2fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:46.859058 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.859041    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:26:46.859096 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.859064    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:26:46.859096 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.859078    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:26:46.860473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.860450    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52f172d8-f26c-4d81-87dc-3fd6c92af2fe" (UID: "52f172d8-f26c-4d81-87dc-3fd6c92af2fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:26:46.860600 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.860575    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h" (OuterVolumeSpecName: "kube-api-access-kkf6h") pod "52f172d8-f26c-4d81-87dc-3fd6c92af2fe" (UID: "52f172d8-f26c-4d81-87dc-3fd6c92af2fe"). InnerVolumeSpecName "kube-api-access-kkf6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:26:46.960108 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.960058    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:26:46.960108 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:46.960078    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkf6h\" (UniqueName: \"kubernetes.io/projected/52f172d8-f26c-4d81-87dc-3fd6c92af2fe-kube-api-access-kkf6h\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:26:47.135460 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:47.135434    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:26:47.139049 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:47.139021    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-qn7xn"]
Jul 02 13:26:47.566302 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:47.566271    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" path="/var/lib/kubelet/pods/52f172d8-f26c-4d81-87dc-3fd6c92af2fe/volumes"
Jul 02 13:26:49.709717 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:49.709679    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:49.710101 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:49.710028    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:26:59.710270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:59.710230    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:26:59.710670 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:26:59.710578    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:27:09.709327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:09.709287    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:27:09.709788 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:09.709629    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:27:19.710078 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:19.710037    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:27:19.710519 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:19.710415    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:27:29.710117 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:29.710073    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:27:29.710519 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:29.710477    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:27:39.710066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:39.710037    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:27:39.710555 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:39.710535    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:27:51.850847 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.850809    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851163    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851174    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851192    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851198    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851214    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="storage-initializer"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851219    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="storage-initializer"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851226    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851231    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851279    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kserve-container"
Jul 02 13:27:51.851281 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851288    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="kube-rbac-proxy"
Jul 02 13:27:51.851694 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.851294    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f172d8-f26c-4d81-87dc-3fd6c92af2fe" containerName="agent"
Jul 02 13:27:51.854001 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.853984    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:51.860038 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.859996    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 02 13:27:51.860156 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.860085    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 02 13:27:51.863082 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.863063    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:27:51.863340 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.863291    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" containerID="cri-o://b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6" gracePeriod=30
Jul 02 13:27:51.863439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.863364    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" containerID="cri-o://028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e" gracePeriod=30
Jul 02 13:27:51.863439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.863370    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" containerID="cri-o://6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8" gracePeriod=30
Jul 02 13:27:51.869866 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.869841    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:27:51.978150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.978119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:51.978283 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.978183    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:51.978283 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:51.978215    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vrv\" (UniqueName: \"kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.078703 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.078675    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.078829 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.078723    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.078829 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.078749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vrv\" (UniqueName: \"kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.079359 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.079339    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.080914 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.080866    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.087846 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.087823    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vrv\" (UniqueName: \"kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv\") pod \"message-dumper-predictor-6575bcfdfd-q68bs\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.098467 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.098446    2552 generic.go:356] "Generic (PLEG): container finished" podID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerID="6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8" exitCode=2
Jul 02 13:27:52.098568 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.098486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerDied","Data":"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"}
Jul 02 13:27:52.164381 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.164332    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:52.298739 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.298718    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:27:52.300485 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:27:52.300457    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9316cb6f_ebea_45d6_9d40_2c11ab6ccb45.slice/crio-d3268fc81993e08f795806ec6cb231862ad64b63ef47acf8b8968409bbef45f0 WatchSource:0}: Error finding container d3268fc81993e08f795806ec6cb231862ad64b63ef47acf8b8968409bbef45f0: Status 404 returned error can't find the container with id d3268fc81993e08f795806ec6cb231862ad64b63ef47acf8b8968409bbef45f0
Jul 02 13:27:52.302299 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:52.302283    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:27:53.103657 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:53.103624    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerStarted","Data":"d3268fc81993e08f795806ec6cb231862ad64b63ef47acf8b8968409bbef45f0"}
Jul 02 13:27:54.110370 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.110339    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerStarted","Data":"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"}
Jul 02 13:27:54.110818 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.110482    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:54.110818 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.110526    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerStarted","Data":"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"}
Jul 02 13:27:54.110818 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.110666    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:54.112237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.112219    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:27:54.139432 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.139389    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" podStartSLOduration=1.798124966 podStartE2EDuration="3.139376646s" totalImagesPullingTime="1.34125168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:27:54.138400013 +0000 UTC m=+785.993792895" watchObservedRunningTime="2026-07-02 13:27:54.139376646 +0000 UTC m=+785.994769529"
Jul 02 13:27:54.704742 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:54.704699    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:27:56.121502 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:56.121473    2552 generic.go:356] "Generic (PLEG): container finished" podID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerID="b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6" exitCode=0
Jul 02 13:27:56.121802 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:56.121527    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerDied","Data":"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"}
Jul 02 13:27:59.704690 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:59.704654    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:27:59.710011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:59.709987    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:27:59.711556 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:27:59.711532    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:01.126003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.125977    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:28:01.874268 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.874239    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:28:01.878147 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.878127    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:01.880598 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.880572    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 02 13:28:01.880598 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.880582    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 02 13:28:01.888463 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:01.888439    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:28:02.058278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.058247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.058278 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.058276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.058454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.058311    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.058454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.058397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.058454 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.058442    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzptz\" (UniqueName: \"kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.159729 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.159670    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.159729 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.159709    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160131 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.159753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzptz\" (UniqueName: \"kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160131 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.159907    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160131 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.159934    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.160137    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160542 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.160526    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.160576 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.160557    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.161989 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.161970    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.167851 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.167829    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzptz\" (UniqueName: \"kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz\") pod \"isvc-logger-predictor-5b7645fdfc-sxvfn\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.189250 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.189217    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:02.317528 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:02.317483    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:28:02.320414 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:28:02.320391    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380ef084_ba90_4549_8261_3d831d7ebe27.slice/crio-ebba8d4063b15a5953f059e8887f899266365668fede62a13be3c68876ae74e8 WatchSource:0}: Error finding container ebba8d4063b15a5953f059e8887f899266365668fede62a13be3c68876ae74e8: Status 404 returned error can't find the container with id ebba8d4063b15a5953f059e8887f899266365668fede62a13be3c68876ae74e8
Jul 02 13:28:03.154362 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:03.154325    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerStarted","Data":"099018ccdc57823d75f83736267c481397b337d41b8d97a76484397445025ccb"}
Jul 02 13:28:03.154527 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:03.154405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerStarted","Data":"ebba8d4063b15a5953f059e8887f899266365668fede62a13be3c68876ae74e8"}
Jul 02 13:28:04.704508 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:04.704470    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:28:04.704928 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:04.704575    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:28:06.170947 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:06.170924    2552 generic.go:356] "Generic (PLEG): container finished" podID="380ef084-ba90-4549-8261-3d831d7ebe27" containerID="099018ccdc57823d75f83736267c481397b337d41b8d97a76484397445025ccb" exitCode=0
Jul 02 13:28:06.171233 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:06.170996    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerDied","Data":"099018ccdc57823d75f83736267c481397b337d41b8d97a76484397445025ccb"}
Jul 02 13:28:07.177780 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:07.177744    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerStarted","Data":"b72c9fd68793f31f7b2ae87bb0feaf0473547e5185b00d777c29081af5abc6c6"}
Jul 02 13:28:07.178259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:07.177855    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerStarted","Data":"5a6e7bec443aa402beed724ef626cbe6f459e11f62abc5de77e0bf60727f5d1b"}
Jul 02 13:28:07.178259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:07.177927    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerStarted","Data":"af2cfae5f912b0729f1762c62bc6e0bacfeed4ac905526d563dc2d5f4958b249"}
Jul 02 13:28:07.178259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:07.178171    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:07.201518 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:07.201474    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podStartSLOduration=2.484681374 podStartE2EDuration="6.201461534s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71678016s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:28:07.199772348 +0000 UTC m=+799.055165253" watchObservedRunningTime="2026-07-02 13:28:07.201461534 +0000 UTC m=+799.056854416"
Jul 02 13:28:08.182940 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:08.182912    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:08.183287 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:08.183015    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:08.184333 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:08.184293    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:08.185003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:08.184976    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:09.187526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:09.187485    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:09.187949 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:09.187854    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:09.705122 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:09.705085    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:28:09.709440 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:09.709414    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:28:09.711066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:09.711042    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:14.191704 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:14.191677    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:28:14.192455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:14.192422    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:14.192593 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:14.192562    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:14.704953 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:14.704922    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:28:19.704628 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:19.704587    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 02 13:28:19.710040 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:19.710019    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Jul 02 13:28:19.710124 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:19.710112    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:28:19.711599 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:19.711574    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:19.711692 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:19.711665    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:28:22.013534 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.013512    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:28:22.117068 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117042    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config\") pod \"337c7777-eac4-4665-80f2-f67913d0e4b8\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") "
Jul 02 13:28:22.117210 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117104    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") pod \"337c7777-eac4-4665-80f2-f67913d0e4b8\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") "
Jul 02 13:28:22.117263 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117220    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc279\" (UniqueName: \"kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279\") pod \"337c7777-eac4-4665-80f2-f67913d0e4b8\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") "
Jul 02 13:28:22.117263 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117247    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert\") pod \"337c7777-eac4-4665-80f2-f67913d0e4b8\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") "
Jul 02 13:28:22.117367 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117269    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location\") pod \"337c7777-eac4-4665-80f2-f67913d0e4b8\" (UID: \"337c7777-eac4-4665-80f2-f67913d0e4b8\") "
Jul 02 13:28:22.117484 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117407    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "337c7777-eac4-4665-80f2-f67913d0e4b8" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:28:22.117608 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117559    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "337c7777-eac4-4665-80f2-f67913d0e4b8" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:28:22.117672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117618    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:28:22.117672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.117619    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "337c7777-eac4-4665-80f2-f67913d0e4b8" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:22.119057 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.119038    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "337c7777-eac4-4665-80f2-f67913d0e4b8" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:28:22.119150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.119117    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279" (OuterVolumeSpecName: "kube-api-access-vc279") pod "337c7777-eac4-4665-80f2-f67913d0e4b8" (UID: "337c7777-eac4-4665-80f2-f67913d0e4b8"). InnerVolumeSpecName "kube-api-access-vc279". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:28:22.218781 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.218758    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337c7777-eac4-4665-80f2-f67913d0e4b8-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:28:22.218781 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.218780    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc279\" (UniqueName: \"kubernetes.io/projected/337c7777-eac4-4665-80f2-f67913d0e4b8-kube-api-access-vc279\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:28:22.218939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.218795    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337c7777-eac4-4665-80f2-f67913d0e4b8-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:28:22.218939 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.218807    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337c7777-eac4-4665-80f2-f67913d0e4b8-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:28:22.248646 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.248620    2552 generic.go:356] "Generic (PLEG): container finished" podID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerID="028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e" exitCode=0
Jul 02 13:28:22.248737 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.248684    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"
Jul 02 13:28:22.248737 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.248709    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerDied","Data":"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"}
Jul 02 13:28:22.248804 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.248782    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c" event={"ID":"337c7777-eac4-4665-80f2-f67913d0e4b8","Type":"ContainerDied","Data":"ba96624925ff829bb3f181355e5080b9aad70233673abdf7fe7e90dfe25e6ad7"}
Jul 02 13:28:22.248841 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.248814    2552 scope.go:117] "RemoveContainer" containerID="028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"
Jul 02 13:28:22.257238 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.257222    2552 scope.go:117] "RemoveContainer" containerID="6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"
Jul 02 13:28:22.264336 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.264316    2552 scope.go:117] "RemoveContainer" containerID="b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"
Jul 02 13:28:22.271556 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.271509    2552 scope.go:117] "RemoveContainer" containerID="7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"
Jul 02 13:28:22.273268 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.273251    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:28:22.279003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.278981    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-z5x5c"]
Jul 02 13:28:22.279368 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.279353    2552 scope.go:117] "RemoveContainer" containerID="028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"
Jul 02 13:28:22.279609 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:28:22.279593    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e\": container with ID starting with 028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e not found: ID does not exist" containerID="028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"
Jul 02 13:28:22.279672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.279615    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e"} err="failed to get container status \"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e\": rpc error: code = NotFound desc = could not find container \"028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e\": container with ID starting with 028f05da4cbdecc919135260791a3060be9c0110142de3dd3f0dd711112d6a4e not found: ID does not exist"
Jul 02 13:28:22.279672 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.279631    2552 scope.go:117] "RemoveContainer" containerID="6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"
Jul 02 13:28:22.279862 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:28:22.279846    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8\": container with ID starting with 6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8 not found: ID does not exist" containerID="6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"
Jul 02 13:28:22.279928 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.279894    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8"} err="failed to get container status \"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8\": rpc error: code = NotFound desc = could not find container \"6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8\": container with ID starting with 6c9e5b1373eea085febd8f7eae6cc9639e40268bbe9878e9d47415b578ad2ed8 not found: ID does not exist"
Jul 02 13:28:22.279928 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.279917    2552 scope.go:117] "RemoveContainer" containerID="b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"
Jul 02 13:28:22.280182 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:28:22.280167    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6\": container with ID starting with b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6 not found: ID does not exist" containerID="b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"
Jul 02 13:28:22.280235 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.280188    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6"} err="failed to get container status \"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6\": rpc error: code = NotFound desc = could not find container \"b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6\": container with ID starting with b33ffa3eaddb35327e8bc4c2ad1762b57d45ac0e05f890324623f0a5b0351fe6 not found: ID does not exist"
Jul 02 13:28:22.280235 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.280201    2552 scope.go:117] "RemoveContainer" containerID="7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"
Jul 02 13:28:22.280437 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:28:22.280416    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250\": container with ID starting with 7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250 not found: ID does not exist" containerID="7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"
Jul 02 13:28:22.280522 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:22.280442    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250"} err="failed to get container status \"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250\": rpc error: code = NotFound desc = could not find container \"7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250\": container with ID starting with 7b6d1b265423506caa3df535d2104c5f556475849b6346dfdb3943ea2c023250 not found: ID does not exist"
Jul 02 13:28:23.566022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:23.565994    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" path="/var/lib/kubelet/pods/337c7777-eac4-4665-80f2-f67913d0e4b8/volumes"
Jul 02 13:28:24.197976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:24.197717    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:24.198372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:24.198338    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:34.193151 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:34.193113    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:34.193601 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:34.193575    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:44.192317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:44.192281    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:44.192750 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:44.192729    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:54.193055 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:54.193015    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:28:54.193467 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:28:54.193445    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:04.193213 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:04.193178    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:29:04.193693 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:04.193667    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:14.193066 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:14.193035    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:14.193464 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:14.193131    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:26.894292 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:26.894262    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-q68bs_9316cb6f-ebea-45d6-9d40-2c11ab6ccb45/kserve-container/0.log"
Jul 02 13:29:27.113614 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.113576    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:29:27.113923 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.113858    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" containerID="cri-o://af2cfae5f912b0729f1762c62bc6e0bacfeed4ac905526d563dc2d5f4958b249" gracePeriod=30
Jul 02 13:29:27.114035 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.113913    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" containerID="cri-o://5a6e7bec443aa402beed724ef626cbe6f459e11f62abc5de77e0bf60727f5d1b" gracePeriod=30
Jul 02 13:29:27.114035 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.113900    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" containerID="cri-o://b72c9fd68793f31f7b2ae87bb0feaf0473547e5185b00d777c29081af5abc6c6" gracePeriod=30
Jul 02 13:29:27.171930 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.171856    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:29:27.172279 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172264    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent"
Jul 02 13:29:27.172323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172281    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent"
Jul 02 13:29:27.172323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172295    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container"
Jul 02 13:29:27.172323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172301    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container"
Jul 02 13:29:27.172323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172310    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="storage-initializer"
Jul 02 13:29:27.172323 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172316    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="storage-initializer"
Jul 02 13:29:27.172473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172328    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy"
Jul 02 13:29:27.172473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172333    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy"
Jul 02 13:29:27.172473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172399    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kube-rbac-proxy"
Jul 02 13:29:27.172473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172407    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="agent"
Jul 02 13:29:27.172473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.172419    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="337c7777-eac4-4665-80f2-f67913d0e4b8" containerName="kserve-container"
Jul 02 13:29:27.175629 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.175612    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.178165 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.178146    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 02 13:29:27.178165 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.178155    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 02 13:29:27.190581 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.190558    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:29:27.220546 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.220515    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.220708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.220573    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.220708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.220619    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8dv\" (UniqueName: \"kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.220708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.220644    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.220708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.220673    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.250677 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.250644    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:29:27.250918 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.250885    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kserve-container" containerID="cri-o://d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a" gracePeriod=30
Jul 02 13:29:27.251031 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.250971    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kube-rbac-proxy" containerID="cri-o://7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c" gracePeriod=30
Jul 02 13:29:27.321370 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321342    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.321483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321438    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.321483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321477    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.321588 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321519    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8dv\" (UniqueName: \"kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.321644 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321620    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.322010 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.321987    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.322204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.322185    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.322267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.322186    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.323796 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.323774    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.330658 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.330640    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8dv\" (UniqueName: \"kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.481697 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.481667    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:29:27.485266 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.485247    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:29:27.562625 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.562594    2552 generic.go:356] "Generic (PLEG): container finished" podID="380ef084-ba90-4549-8261-3d831d7ebe27" containerID="5a6e7bec443aa402beed724ef626cbe6f459e11f62abc5de77e0bf60727f5d1b" exitCode=2
Jul 02 13:29:27.564441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.564417    2552 generic.go:356] "Generic (PLEG): container finished" podID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerID="7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c" exitCode=2
Jul 02 13:29:27.564441 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.564438    2552 generic.go:356] "Generic (PLEG): container finished" podID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerID="d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a" exitCode=2
Jul 02 13:29:27.564599 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.564493    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"
Jul 02 13:29:27.576741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.576723    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerDied","Data":"5a6e7bec443aa402beed724ef626cbe6f459e11f62abc5de77e0bf60727f5d1b"}
Jul 02 13:29:27.576906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.576890    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerDied","Data":"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"}
Jul 02 13:29:27.577037 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.577023    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerDied","Data":"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"}
Jul 02 13:29:27.577147 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.577108    2552 scope.go:117] "RemoveContainer" containerID="7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"
Jul 02 13:29:27.577336 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.577142    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs" event={"ID":"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45","Type":"ContainerDied","Data":"d3268fc81993e08f795806ec6cb231862ad64b63ef47acf8b8968409bbef45f0"}
Jul 02 13:29:27.587752 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.587731    2552 scope.go:117] "RemoveContainer" containerID="d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"
Jul 02 13:29:27.595859 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.595840    2552 scope.go:117] "RemoveContainer" containerID="7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"
Jul 02 13:29:27.596310 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:29:27.596163    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c\": container with ID starting with 7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c not found: ID does not exist" containerID="7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"
Jul 02 13:29:27.596310 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596198    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"} err="failed to get container status \"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c\": rpc error: code = NotFound desc = could not find container \"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c\": container with ID starting with 7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c not found: ID does not exist"
Jul 02 13:29:27.596310 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596225    2552 scope.go:117] "RemoveContainer" containerID="d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"
Jul 02 13:29:27.596565 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:29:27.596536    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a\": container with ID starting with d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a not found: ID does not exist" containerID="d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"
Jul 02 13:29:27.596622 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596564    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"} err="failed to get container status \"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a\": rpc error: code = NotFound desc = could not find container \"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a\": container with ID starting with d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a not found: ID does not exist"
Jul 02 13:29:27.596622 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596603    2552 scope.go:117] "RemoveContainer" containerID="7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"
Jul 02 13:29:27.596912 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596891    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c"} err="failed to get container status \"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c\": rpc error: code = NotFound desc = could not find container \"7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c\": container with ID starting with 7395f1e8aad65a794880060c2ce740d96c7f0b817843cb14e0024b2c6249a12c not found: ID does not exist"
Jul 02 13:29:27.597003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.596913    2552 scope.go:117] "RemoveContainer" containerID="d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"
Jul 02 13:29:27.597180 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.597161    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a"} err="failed to get container status \"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a\": rpc error: code = NotFound desc = could not find container \"d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a\": container with ID starting with d88c9b0e76c1bc6ad33fb12ed6a5250d9f86154528c84aadf04260e3ad92376a not found: ID does not exist"
Jul 02 13:29:27.624688 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.624663    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls\") pod \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") "
Jul 02 13:29:27.624848 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.624743    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config\") pod \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") "
Jul 02 13:29:27.624848 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.624834    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4vrv\" (UniqueName: \"kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv\") pod \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\" (UID: \"9316cb6f-ebea-45d6-9d40-2c11ab6ccb45\") "
Jul 02 13:29:27.625247 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.625198    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" (UID: "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:27.626833 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.626813    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" (UID: "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:27.626833 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.626810    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv" (OuterVolumeSpecName: "kube-api-access-g4vrv") pod "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" (UID: "9316cb6f-ebea-45d6-9d40-2c11ab6ccb45"). InnerVolumeSpecName "kube-api-access-g4vrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:27.627015 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.626995    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:29:27.628948 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:29:27.628928    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fae0d0a_8aab_4537_a7ef_71b83297e7fe.slice/crio-68100ca04c7ec5221d572064fe37ef8edc91ac5caf9acabc13329cea8cc7847e WatchSource:0}: Error finding container 68100ca04c7ec5221d572064fe37ef8edc91ac5caf9acabc13329cea8cc7847e: Status 404 returned error can't find the container with id 68100ca04c7ec5221d572064fe37ef8edc91ac5caf9acabc13329cea8cc7847e
Jul 02 13:29:27.726479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.726402    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vrv\" (UniqueName: \"kubernetes.io/projected/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-api-access-g4vrv\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:27.726479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.726440    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:27.726479 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.726450    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:27.886367 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.886338    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:29:27.890342 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:27.890316    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-q68bs"]
Jul 02 13:29:28.570938 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:28.570905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerStarted","Data":"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"}
Jul 02 13:29:28.571395 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:28.570955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerStarted","Data":"68100ca04c7ec5221d572064fe37ef8edc91ac5caf9acabc13329cea8cc7847e"}
Jul 02 13:29:29.187986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:29.187941    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:29.566230 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:29.566198    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" path="/var/lib/kubelet/pods/9316cb6f-ebea-45d6-9d40-2c11ab6ccb45/volumes"
Jul 02 13:29:31.585903 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:31.585821    2552 generic.go:356] "Generic (PLEG): container finished" podID="380ef084-ba90-4549-8261-3d831d7ebe27" containerID="af2cfae5f912b0729f1762c62bc6e0bacfeed4ac905526d563dc2d5f4958b249" exitCode=0
Jul 02 13:29:31.585903 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:31.585891    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerDied","Data":"af2cfae5f912b0729f1762c62bc6e0bacfeed4ac905526d563dc2d5f4958b249"}
Jul 02 13:29:34.188599 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:34.188560    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:34.192914 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:34.192890    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:29:34.193261 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:34.193242    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:39.188163 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:39.188126    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:39.188583 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:39.188230    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:44.188409 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:44.188323    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:44.192861 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:44.192833    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:29:44.193147 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:44.193124    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:49.187783 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:49.187744    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:54.188153 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:54.188117    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 02 13:29:54.192422 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:54.192388    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 02 13:29:54.192565 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:54.192508    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:54.192796 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:54.192779    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:54.192893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:54.192862    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:57.708552 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.708523    2552 generic.go:356] "Generic (PLEG): container finished" podID="380ef084-ba90-4549-8261-3d831d7ebe27" containerID="b72c9fd68793f31f7b2ae87bb0feaf0473547e5185b00d777c29081af5abc6c6" exitCode=0
Jul 02 13:29:57.708941 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.708595    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerDied","Data":"b72c9fd68793f31f7b2ae87bb0feaf0473547e5185b00d777c29081af5abc6c6"}
Jul 02 13:29:57.710206 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.710185    2552 generic.go:356] "Generic (PLEG): container finished" podID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerID="479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559" exitCode=0
Jul 02 13:29:57.710337 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.710228    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerDied","Data":"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"}
Jul 02 13:29:57.766276 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.766253    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:57.863039 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863010    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config\") pod \"380ef084-ba90-4549-8261-3d831d7ebe27\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") "
Jul 02 13:29:57.863178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863045    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location\") pod \"380ef084-ba90-4549-8261-3d831d7ebe27\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") "
Jul 02 13:29:57.863178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863064    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert\") pod \"380ef084-ba90-4549-8261-3d831d7ebe27\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") "
Jul 02 13:29:57.863178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863101    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzptz\" (UniqueName: \"kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz\") pod \"380ef084-ba90-4549-8261-3d831d7ebe27\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") "
Jul 02 13:29:57.863178 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863129    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls\") pod \"380ef084-ba90-4549-8261-3d831d7ebe27\" (UID: \"380ef084-ba90-4549-8261-3d831d7ebe27\") "
Jul 02 13:29:57.863425 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863395    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "380ef084-ba90-4549-8261-3d831d7ebe27" (UID: "380ef084-ba90-4549-8261-3d831d7ebe27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:57.863425 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863413    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "380ef084-ba90-4549-8261-3d831d7ebe27" (UID: "380ef084-ba90-4549-8261-3d831d7ebe27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:57.863573 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863473    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "380ef084-ba90-4549-8261-3d831d7ebe27" (UID: "380ef084-ba90-4549-8261-3d831d7ebe27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:57.863573 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863547    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:57.863573 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863566    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380ef084-ba90-4549-8261-3d831d7ebe27-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:57.863695 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.863580    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380ef084-ba90-4549-8261-3d831d7ebe27-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:57.865108 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.865080    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz" (OuterVolumeSpecName: "kube-api-access-zzptz") pod "380ef084-ba90-4549-8261-3d831d7ebe27" (UID: "380ef084-ba90-4549-8261-3d831d7ebe27"). InnerVolumeSpecName "kube-api-access-zzptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:57.865207 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.865130    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "380ef084-ba90-4549-8261-3d831d7ebe27" (UID: "380ef084-ba90-4549-8261-3d831d7ebe27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:57.964496 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.964480    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzptz\" (UniqueName: \"kubernetes.io/projected/380ef084-ba90-4549-8261-3d831d7ebe27-kube-api-access-zzptz\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:57.964496 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:57.964499    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380ef084-ba90-4549-8261-3d831d7ebe27-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:29:58.720023 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:58.719986    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"
Jul 02 13:29:58.720439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:58.719992    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn" event={"ID":"380ef084-ba90-4549-8261-3d831d7ebe27","Type":"ContainerDied","Data":"ebba8d4063b15a5953f059e8887f899266365668fede62a13be3c68876ae74e8"}
Jul 02 13:29:58.720439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:58.720151    2552 scope.go:117] "RemoveContainer" containerID="b72c9fd68793f31f7b2ae87bb0feaf0473547e5185b00d777c29081af5abc6c6"
Jul 02 13:29:58.742769 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:58.742738    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:29:58.747470 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:58.747448    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-sxvfn"]
Jul 02 13:29:59.567166 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:59.567135    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" path="/var/lib/kubelet/pods/380ef084-ba90-4549-8261-3d831d7ebe27/volumes"
Jul 02 13:29:59.750311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:59.750081    2552 scope.go:117] "RemoveContainer" containerID="5a6e7bec443aa402beed724ef626cbe6f459e11f62abc5de77e0bf60727f5d1b"
Jul 02 13:29:59.846256 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:59.846236    2552 scope.go:117] "RemoveContainer" containerID="af2cfae5f912b0729f1762c62bc6e0bacfeed4ac905526d563dc2d5f4958b249"
Jul 02 13:29:59.933845 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:29:59.933814    2552 scope.go:117] "RemoveContainer" containerID="099018ccdc57823d75f83736267c481397b337d41b8d97a76484397445025ccb"
Jul 02 13:32:16.423937 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.423840    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerStarted","Data":"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"}
Jul 02 13:32:16.423937 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.423920    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerStarted","Data":"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"}
Jul 02 13:32:16.424459 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.423944    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:16.424459 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.423964    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:16.425328 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.425306    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:32:16.453664 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:16.453619    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podStartSLOduration=1.440146323 podStartE2EDuration="2m49.453606157s" totalImagesPullingTime="2m18.345658928s" totalInitContainerRuntime="29.667800906s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:32:16.451718043 +0000 UTC m=+1048.307110953" watchObservedRunningTime="2026-07-02 13:32:16.453606157 +0000 UTC m=+1048.308999040"
Jul 02 13:32:17.428201 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:17.428161    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:32:22.433901 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:22.433855    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:22.434503 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:22.434419    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:32:32.435042 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:32.435014    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:37.379011 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:37.378969    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:32:37.379469 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:37.379240    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container" containerID="cri-o://c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17" gracePeriod=30
Jul 02 13:32:37.379544 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:37.379456    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kube-rbac-proxy" containerID="cri-o://e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced" gracePeriod=30
Jul 02 13:32:37.429202 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:37.429157    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 02 13:32:38.469134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.469096    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:38.515951 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.515853    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g8dv\" (UniqueName: \"kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv\") pod \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") "
Jul 02 13:32:38.515951 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.515931    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls\") pod \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") "
Jul 02 13:32:38.516204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.515957    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert\") pod \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") "
Jul 02 13:32:38.516204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.516051    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location\") pod \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") "
Jul 02 13:32:38.516204 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.516091    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config\") pod \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\" (UID: \"6fae0d0a-8aab-4537-a7ef-71b83297e7fe\") "
Jul 02 13:32:38.516414 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.516394    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fae0d0a-8aab-4537-a7ef-71b83297e7fe" (UID: "6fae0d0a-8aab-4537-a7ef-71b83297e7fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:32:38.516640 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.516610    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fae0d0a-8aab-4537-a7ef-71b83297e7fe" (UID: "6fae0d0a-8aab-4537-a7ef-71b83297e7fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:32:38.518305 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.518263    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv" (OuterVolumeSpecName: "kube-api-access-6g8dv") pod "6fae0d0a-8aab-4537-a7ef-71b83297e7fe" (UID: "6fae0d0a-8aab-4537-a7ef-71b83297e7fe"). InnerVolumeSpecName "kube-api-access-6g8dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:32:38.519058 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.519032    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fae0d0a-8aab-4537-a7ef-71b83297e7fe" (UID: "6fae0d0a-8aab-4537-a7ef-71b83297e7fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:32:38.521209 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.521184    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fae0d0a-8aab-4537-a7ef-71b83297e7fe" (UID: "6fae0d0a-8aab-4537-a7ef-71b83297e7fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:38.523973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.523946    2552 generic.go:356] "Generic (PLEG): container finished" podID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerID="e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced" exitCode=2
Jul 02 13:32:38.523973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.523968    2552 generic.go:356] "Generic (PLEG): container finished" podID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerID="c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17" exitCode=0
Jul 02 13:32:38.524132 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.524018    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"
Jul 02 13:32:38.524132 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.524023    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerDied","Data":"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"}
Jul 02 13:32:38.524132 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.524097    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerDied","Data":"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"}
Jul 02 13:32:38.524132 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.524128    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj" event={"ID":"6fae0d0a-8aab-4537-a7ef-71b83297e7fe","Type":"ContainerDied","Data":"68100ca04c7ec5221d572064fe37ef8edc91ac5caf9acabc13329cea8cc7847e"}
Jul 02 13:32:38.524393 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.524160    2552 scope.go:117] "RemoveContainer" containerID="e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"
Jul 02 13:32:38.536372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.536349    2552 scope.go:117] "RemoveContainer" containerID="c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"
Jul 02 13:32:38.544460 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.544439    2552 scope.go:117] "RemoveContainer" containerID="479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"
Jul 02 13:32:38.548908 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.548867    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:32:38.552611 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.552583    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2lhrj"]
Jul 02 13:32:38.554031 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554014    2552 scope.go:117] "RemoveContainer" containerID="e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"
Jul 02 13:32:38.554353 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:32:38.554331    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced\": container with ID starting with e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced not found: ID does not exist" containerID="e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"
Jul 02 13:32:38.554404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554363    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"} err="failed to get container status \"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced\": rpc error: code = NotFound desc = could not find container \"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced\": container with ID starting with e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced not found: ID does not exist"
Jul 02 13:32:38.554404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554380    2552 scope.go:117] "RemoveContainer" containerID="c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"
Jul 02 13:32:38.554651 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:32:38.554632    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17\": container with ID starting with c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17 not found: ID does not exist" containerID="c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"
Jul 02 13:32:38.554688 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554658    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"} err="failed to get container status \"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17\": rpc error: code = NotFound desc = could not find container \"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17\": container with ID starting with c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17 not found: ID does not exist"
Jul 02 13:32:38.554688 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554675    2552 scope.go:117] "RemoveContainer" containerID="479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"
Jul 02 13:32:38.554903 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:32:38.554885    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559\": container with ID starting with 479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559 not found: ID does not exist" containerID="479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"
Jul 02 13:32:38.554952 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554908    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"} err="failed to get container status \"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559\": rpc error: code = NotFound desc = could not find container \"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559\": container with ID starting with 479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559 not found: ID does not exist"
Jul 02 13:32:38.554952 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.554922    2552 scope.go:117] "RemoveContainer" containerID="e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"
Jul 02 13:32:38.555202 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.555182    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced"} err="failed to get container status \"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced\": rpc error: code = NotFound desc = could not find container \"e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced\": container with ID starting with e140396a00edd1656466e055c8f863f9b2fdcc4dfc8d48011e20924b8f3daced not found: ID does not exist"
Jul 02 13:32:38.555246 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.555204    2552 scope.go:117] "RemoveContainer" containerID="c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"
Jul 02 13:32:38.555449 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.555430    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17"} err="failed to get container status \"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17\": rpc error: code = NotFound desc = could not find container \"c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17\": container with ID starting with c97ae61e571ab64b6768304dcbaae99a3e404de1d47782252de4fc01475a2c17 not found: ID does not exist"
Jul 02 13:32:38.555496 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.555451    2552 scope.go:117] "RemoveContainer" containerID="479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"
Jul 02 13:32:38.555650 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.555633    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559"} err="failed to get container status \"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559\": rpc error: code = NotFound desc = could not find container \"479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559\": container with ID starting with 479be9b3d6bd8a270d7b4b5d2c38a85960317be430e55a4f5c635134245b4559 not found: ID does not exist"
Jul 02 13:32:38.617498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.617458    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8dv\" (UniqueName: \"kubernetes.io/projected/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-api-access-6g8dv\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:32:38.617498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.617493    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:32:38.617498 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.617504    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:32:38.617722 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.617514    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:32:38.617722 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:38.617523    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fae0d0a-8aab-4537-a7ef-71b83297e7fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:32:39.566285 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:39.566253    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" path="/var/lib/kubelet/pods/6fae0d0a-8aab-4537-a7ef-71b83297e7fe/volumes"
Jul 02 13:32:42.688318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688288    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:32:42.688675 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688635    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688675 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688646    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688675 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688657    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container"
Jul 02 13:32:42.688675 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688666    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688678    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688685    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688692    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688697    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688709    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kserve-container"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688714    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kserve-container"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688720    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688726    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688733    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="storage-initializer"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688739    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="storage-initializer"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688754    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688760    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688767    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="storage-initializer"
Jul 02 13:32:42.688805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688772    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="storage-initializer"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688820    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kube-rbac-proxy"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688832    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kserve-container"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688840    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kserve-container"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688846    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9316cb6f-ebea-45d6-9d40-2c11ab6ccb45" containerName="kube-rbac-proxy"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688852    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fae0d0a-8aab-4537-a7ef-71b83297e7fe" containerName="kube-rbac-proxy"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688858    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="kserve-container"
Jul 02 13:32:42.689259 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.688865    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="380ef084-ba90-4549-8261-3d831d7ebe27" containerName="agent"
Jul 02 13:32:42.717155 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.717130    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:32:42.717297 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.717240    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.719968 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.719852    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:32:42.719968 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.719895    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 02 13:32:42.719968 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.719946    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f7h2p\""
Jul 02 13:32:42.719968 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.719958    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:32:42.720280 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.719898    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:32:42.720280 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.720117    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 02 13:32:42.720913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.720897    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:32:42.857118 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.857094    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lp7\" (UniqueName: \"kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.857251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.857136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.857251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.857162    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.857251 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.857231    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.857370 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.857260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958181 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958192    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958216    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958278    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lp7\" (UniqueName: \"kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958404 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:32:42.958303    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 02 13:32:42.958404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958326    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958404 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:32:42.958379    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls podName:5b8dc7b3-5163-424d-845b-ec3f23356511 nodeName:}" failed. No retries permitted until 2026-07-02 13:32:43.458358382 +0000 UTC m=+1075.313751245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-tm7hb" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 02 13:32:42.958664 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958644    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.958955 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958938    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.959003 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.958955    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:42.971929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:42.971845    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lp7\" (UniqueName: \"kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:43.461844 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:43.461766    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:43.464137 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:43.464115    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-tm7hb\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:43.627937 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:43.627903    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:43.772511 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:43.772389    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:32:43.775259 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:32:43.775229    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8dc7b3_5163_424d_845b_ec3f23356511.slice/crio-d74553549759e29fa64974dfe0a6689fb5315d90dc9396cc5d993e1face463b5 WatchSource:0}: Error finding container d74553549759e29fa64974dfe0a6689fb5315d90dc9396cc5d993e1face463b5: Status 404 returned error can't find the container with id d74553549759e29fa64974dfe0a6689fb5315d90dc9396cc5d993e1face463b5
Jul 02 13:32:44.553743 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:44.553709    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerStarted","Data":"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"}
Jul 02 13:32:44.554016 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:44.553784    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerStarted","Data":"d74553549759e29fa64974dfe0a6689fb5315d90dc9396cc5d993e1face463b5"}
Jul 02 13:32:49.579274 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:49.579241    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerID="2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282" exitCode=0
Jul 02 13:32:49.579641 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:49.579311    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerDied","Data":"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"}
Jul 02 13:32:50.585327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:50.585294    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerStarted","Data":"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"}
Jul 02 13:32:50.585692 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:50.585353    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerStarted","Data":"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"}
Jul 02 13:32:50.585692 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:50.585540    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:50.611328 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:50.611275    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" podStartSLOduration=3.85668943 podStartE2EDuration="8.61125975s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.75457032s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:32:50.608234549 +0000 UTC m=+1082.463627433" watchObservedRunningTime="2026-07-02 13:32:50.61125975 +0000 UTC m=+1082.466652633"
Jul 02 13:32:51.589314 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:51.589282    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:32:57.599397 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:32:57.599369    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:34:25.566929 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:25.566823    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:34:32.797304 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:32.797270    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:34:32.797728 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:32.797581    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kserve-container" containerID="cri-o://2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5" gracePeriod=30
Jul 02 13:34:32.797728 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:32.797598    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kube-rbac-proxy" containerID="cri-o://834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295" gracePeriod=30
Jul 02 13:34:33.026964 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.026928    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerID="834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295" exitCode=2
Jul 02 13:34:33.027182 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.026994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerDied","Data":"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"}
Jul 02 13:34:33.642500 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.642479    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:34:33.733465 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733439    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") pod \"5b8dc7b3-5163-424d-845b-ec3f23356511\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") "
Jul 02 13:34:33.733605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733483    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8lp7\" (UniqueName: \"kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7\") pod \"5b8dc7b3-5163-424d-845b-ec3f23356511\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") "
Jul 02 13:34:33.733605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733512    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert\") pod \"5b8dc7b3-5163-424d-845b-ec3f23356511\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") "
Jul 02 13:34:33.733605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733537    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location\") pod \"5b8dc7b3-5163-424d-845b-ec3f23356511\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") "
Jul 02 13:34:33.733605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733580    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config\") pod \"5b8dc7b3-5163-424d-845b-ec3f23356511\" (UID: \"5b8dc7b3-5163-424d-845b-ec3f23356511\") "
Jul 02 13:34:33.733959 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733933    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b8dc7b3-5163-424d-845b-ec3f23356511" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:34:33.734050 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.733963    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b8dc7b3-5163-424d-845b-ec3f23356511" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:34:33.735513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.735488    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7" (OuterVolumeSpecName: "kube-api-access-l8lp7") pod "5b8dc7b3-5163-424d-845b-ec3f23356511" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511"). InnerVolumeSpecName "kube-api-access-l8lp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:34:33.735513 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.735502    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b8dc7b3-5163-424d-845b-ec3f23356511" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:34:33.738102 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.738082    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b8dc7b3-5163-424d-845b-ec3f23356511" (UID: "5b8dc7b3-5163-424d-845b-ec3f23356511"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:34:33.835118 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.835099    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8dc7b3-5163-424d-845b-ec3f23356511-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:34:33.835118 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.835117    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lp7\" (UniqueName: \"kubernetes.io/projected/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-api-access-l8lp7\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:34:33.835462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.835129    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:34:33.835462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.835138    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b8dc7b3-5163-424d-845b-ec3f23356511-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:34:33.835462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:33.835146    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8dc7b3-5163-424d-845b-ec3f23356511-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:34:34.032764 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.032690    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerID="2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5" exitCode=0
Jul 02 13:34:34.032764 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.032742    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"
Jul 02 13:34:34.032936 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.032739    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerDied","Data":"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"}
Jul 02 13:34:34.032936 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.032891    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb" event={"ID":"5b8dc7b3-5163-424d-845b-ec3f23356511","Type":"ContainerDied","Data":"d74553549759e29fa64974dfe0a6689fb5315d90dc9396cc5d993e1face463b5"}
Jul 02 13:34:34.032936 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.032924    2552 scope.go:117] "RemoveContainer" containerID="834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"
Jul 02 13:34:34.041637 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.041619    2552 scope.go:117] "RemoveContainer" containerID="2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"
Jul 02 13:34:34.049270 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.049254    2552 scope.go:117] "RemoveContainer" containerID="2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"
Jul 02 13:34:34.055618 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.055593    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:34:34.057143 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057122    2552 scope.go:117] "RemoveContainer" containerID="834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"
Jul 02 13:34:34.057361 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:34:34.057344    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295\": container with ID starting with 834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295 not found: ID does not exist" containerID="834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"
Jul 02 13:34:34.057413 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057368    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295"} err="failed to get container status \"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295\": rpc error: code = NotFound desc = could not find container \"834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295\": container with ID starting with 834c7df02944d0f4984367fa0d9aab4efd9e43d438d7a3048a82d34a5d293295 not found: ID does not exist"
Jul 02 13:34:34.057413 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057390    2552 scope.go:117] "RemoveContainer" containerID="2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"
Jul 02 13:34:34.057604 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:34:34.057587    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5\": container with ID starting with 2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5 not found: ID does not exist" containerID="2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"
Jul 02 13:34:34.057646 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057619    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5"} err="failed to get container status \"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5\": rpc error: code = NotFound desc = could not find container \"2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5\": container with ID starting with 2e0f1678accbcdf4d944a723cb0f3026f6e49ef62d651572928da616b354c9c5 not found: ID does not exist"
Jul 02 13:34:34.057646 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057637    2552 scope.go:117] "RemoveContainer" containerID="2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"
Jul 02 13:34:34.057826 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:34:34.057811    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282\": container with ID starting with 2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282 not found: ID does not exist" containerID="2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"
Jul 02 13:34:34.057893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.057831    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282"} err="failed to get container status \"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282\": rpc error: code = NotFound desc = could not find container \"2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282\": container with ID starting with 2ad4d7f3b25ef00fb04b47769838591770645337fbc80ea57219061cb40b1282 not found: ID does not exist"
Jul 02 13:34:34.060516 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:34.060496    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-tm7hb"]
Jul 02 13:34:35.566411 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:35.566378    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" path="/var/lib/kubelet/pods/5b8dc7b3-5163-424d-845b-ec3f23356511/volumes"
Jul 02 13:34:38.113125 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113085    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113446    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="storage-initializer"
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113457    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="storage-initializer"
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113467    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kserve-container"
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113473    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kserve-container"
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113491    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kube-rbac-proxy"
Jul 02 13:34:38.113515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113496    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kube-rbac-proxy"
Jul 02 13:34:38.113706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113564    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kube-rbac-proxy"
Jul 02 13:34:38.113706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.113572    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8dc7b3-5163-424d-845b-ec3f23356511" containerName="kserve-container"
Jul 02 13:34:38.117192 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.117174    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.120462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120430    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 02 13:34:38.120462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120454    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:34:38.120691 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120438    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f7h2p\""
Jul 02 13:34:38.120691 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120493    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:34:38.120790 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120770    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:34:38.120910 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120786    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 02 13:34:38.120910 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.120802    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:34:38.137669 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.137625    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:34:38.176605 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.176570    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.176794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.176620    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.176794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.176729    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.176794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.176774    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.176933 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.176828    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchjb\" (UniqueName: \"kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.277906 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.277828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278099 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.277950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278099 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.277990    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278099 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.278024    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchjb\" (UniqueName: \"kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278099 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.278057    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278459 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.278436    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278693 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.278673    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.278763 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.278721    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.280359 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.280335    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.287455 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.287429    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchjb\" (UniqueName: \"kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.428347 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.428243    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:38.573406 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:38.573347    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:34:38.575739 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:34:38.575712    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150b8077_8d96_4fbd_8e62_65d6e8fb6b62.slice/crio-e8882082dc3a7194c6ea249d827464f69e23376c20cb79d0f9a5648fac0eabbc WatchSource:0}: Error finding container e8882082dc3a7194c6ea249d827464f69e23376c20cb79d0f9a5648fac0eabbc: Status 404 returned error can't find the container with id e8882082dc3a7194c6ea249d827464f69e23376c20cb79d0f9a5648fac0eabbc
Jul 02 13:34:39.057305 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:39.057265    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerStarted","Data":"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"}
Jul 02 13:34:39.057595 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:39.057335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerStarted","Data":"e8882082dc3a7194c6ea249d827464f69e23376c20cb79d0f9a5648fac0eabbc"}
Jul 02 13:34:44.081592 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:44.081558    2552 generic.go:356] "Generic (PLEG): container finished" podID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerID="3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750" exitCode=0
Jul 02 13:34:44.081986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:44.081629    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerDied","Data":"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"}
Jul 02 13:34:45.088759 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:45.088722    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerStarted","Data":"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"}
Jul 02 13:34:45.089225 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:45.088786    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerStarted","Data":"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"}
Jul 02 13:34:45.089225 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:45.088991    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:45.089225 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:45.089173    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:34:45.113642 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:45.113599    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" podStartSLOduration=2.461388671 podStartE2EDuration="7.11358688s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.652198209s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:34:45.111376763 +0000 UTC m=+1196.966769671" watchObservedRunningTime="2026-07-02 13:34:45.11358688 +0000 UTC m=+1196.968979763"
Jul 02 13:34:51.099667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:34:51.099632    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:36:22.564167 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:22.564137    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:36:28.131022 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.130986    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:36:28.131464 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.131259    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kserve-container" containerID="cri-o://772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678" gracePeriod=30
Jul 02 13:36:28.131464 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.131298    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kube-rbac-proxy" containerID="cri-o://60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c" gracePeriod=30
Jul 02 13:36:28.532559 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.532526    2552 generic.go:356] "Generic (PLEG): container finished" podID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerID="60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c" exitCode=2
Jul 02 13:36:28.532735 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.532599    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerDied","Data":"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"}
Jul 02 13:36:28.973153 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:28.973126    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:36:29.077548 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.077520    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls\") pod \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") "
Jul 02 13:36:29.077749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.077565    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rchjb\" (UniqueName: \"kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb\") pod \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") "
Jul 02 13:36:29.077749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.077586    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config\") pod \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") "
Jul 02 13:36:29.077749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.077614    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location\") pod \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") "
Jul 02 13:36:29.077749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.077663    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert\") pod \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\" (UID: \"150b8077-8d96-4fbd-8e62-65d6e8fb6b62\") "
Jul 02 13:36:29.078075 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.078047    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "150b8077-8d96-4fbd-8e62-65d6e8fb6b62" (UID: "150b8077-8d96-4fbd-8e62-65d6e8fb6b62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:36:29.078129 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.078112    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "150b8077-8d96-4fbd-8e62-65d6e8fb6b62" (UID: "150b8077-8d96-4fbd-8e62-65d6e8fb6b62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:36:29.079681 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.079652    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "150b8077-8d96-4fbd-8e62-65d6e8fb6b62" (UID: "150b8077-8d96-4fbd-8e62-65d6e8fb6b62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:36:29.079793 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.079742    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb" (OuterVolumeSpecName: "kube-api-access-rchjb") pod "150b8077-8d96-4fbd-8e62-65d6e8fb6b62" (UID: "150b8077-8d96-4fbd-8e62-65d6e8fb6b62"). InnerVolumeSpecName "kube-api-access-rchjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:36:29.082447 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.082424    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "150b8077-8d96-4fbd-8e62-65d6e8fb6b62" (UID: "150b8077-8d96-4fbd-8e62-65d6e8fb6b62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:36:29.178318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.178246    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:36:29.178318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.178272    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rchjb\" (UniqueName: \"kubernetes.io/projected/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-api-access-rchjb\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:36:29.178318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.178282    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:36:29.178318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.178293    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:36:29.178318 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.178302    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150b8077-8d96-4fbd-8e62-65d6e8fb6b62-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:36:29.539090 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.539059    2552 generic.go:356] "Generic (PLEG): container finished" podID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerID="772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678" exitCode=0
Jul 02 13:36:29.539253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.539125    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"
Jul 02 13:36:29.539253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.539147    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerDied","Data":"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"}
Jul 02 13:36:29.539253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.539211    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2" event={"ID":"150b8077-8d96-4fbd-8e62-65d6e8fb6b62","Type":"ContainerDied","Data":"e8882082dc3a7194c6ea249d827464f69e23376c20cb79d0f9a5648fac0eabbc"}
Jul 02 13:36:29.539253 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.539240    2552 scope.go:117] "RemoveContainer" containerID="60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"
Jul 02 13:36:29.548437 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.548417    2552 scope.go:117] "RemoveContainer" containerID="772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"
Jul 02 13:36:29.555746 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.555728    2552 scope.go:117] "RemoveContainer" containerID="3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"
Jul 02 13:36:29.563427 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.563411    2552 scope.go:117] "RemoveContainer" containerID="60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"
Jul 02 13:36:29.563709 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:36:29.563680    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c\": container with ID starting with 60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c not found: ID does not exist" containerID="60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"
Jul 02 13:36:29.563812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.563718    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c"} err="failed to get container status \"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c\": rpc error: code = NotFound desc = could not find container \"60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c\": container with ID starting with 60fa77b8d2cf1fe1e6263c05ac051b39bb863d28d31c1db9ca6e40790f319c8c not found: ID does not exist"
Jul 02 13:36:29.563812 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.563742    2552 scope.go:117] "RemoveContainer" containerID="772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"
Jul 02 13:36:29.564113 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:36:29.564087    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678\": container with ID starting with 772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678 not found: ID does not exist" containerID="772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"
Jul 02 13:36:29.564215 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.564121    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678"} err="failed to get container status \"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678\": rpc error: code = NotFound desc = could not find container \"772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678\": container with ID starting with 772e10407d02e1d2088ba5666e39a51e404766ca8a7204151b77c4fdeae14678 not found: ID does not exist"
Jul 02 13:36:29.564215 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.564145    2552 scope.go:117] "RemoveContainer" containerID="3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"
Jul 02 13:36:29.565033 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:36:29.564959    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750\": container with ID starting with 3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750 not found: ID does not exist" containerID="3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"
Jul 02 13:36:29.565033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.565009    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750"} err="failed to get container status \"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750\": rpc error: code = NotFound desc = could not find container \"3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750\": container with ID starting with 3688488b91645e0449eda5549154d7c9ebebea4dd5b24d1f1d5dc74b59087750 not found: ID does not exist"
Jul 02 13:36:29.567842 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.567820    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:36:29.567970 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:29.567849    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-pw7h2"]
Jul 02 13:36:31.565720 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:31.565686    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" path="/var/lib/kubelet/pods/150b8077-8d96-4fbd-8e62-65d6e8fb6b62/volumes"
Jul 02 13:36:33.281630 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.281596    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.281963    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kube-rbac-proxy"
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.281975    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kube-rbac-proxy"
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.281992    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="storage-initializer"
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.281998    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="storage-initializer"
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.282005    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kserve-container"
Jul 02 13:36:33.282053 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.282010    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kserve-container"
Jul 02 13:36:33.282249 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.282071    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kube-rbac-proxy"
Jul 02 13:36:33.282249 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.282079    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="150b8077-8d96-4fbd-8e62-65d6e8fb6b62" containerName="kserve-container"
Jul 02 13:36:33.287040 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.287019    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.289638 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.289618    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 02 13:36:33.289772 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.289698    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:36:33.289831 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.289773    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:36:33.289948 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.289926    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 02 13:36:33.290017 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.289938    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:36:33.290590 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.290576    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:36:33.290632 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.290616    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f7h2p\""
Jul 02 13:36:33.296406 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.296381    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:36:33.315690 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.315664    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.315794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.315719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.315794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.315782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.315917 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.315839    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.315917 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.315864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6tn\" (UniqueName: \"kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.416971 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.416926    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.416971 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.416974    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417180 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.416997    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417180 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.417027    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417180 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.417045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6tn\" (UniqueName: \"kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417369 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.417347    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417676 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.417638    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.417805 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.417755    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.419526 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.419508    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.424667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.424647    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6tn\" (UniqueName: \"kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.598604 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.598519    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:33.728439 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:33.728409    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:36:33.731333 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:36:33.731306    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9015751b_c0ff_400e_b866_8774befb8951.slice/crio-b4e87661505ff45aabda0405caf3e3111d7a5d020c33151cffc187839d0ccb28 WatchSource:0}: Error finding container b4e87661505ff45aabda0405caf3e3111d7a5d020c33151cffc187839d0ccb28: Status 404 returned error can't find the container with id b4e87661505ff45aabda0405caf3e3111d7a5d020c33151cffc187839d0ccb28
Jul 02 13:36:34.562785 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:34.562742    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerStarted","Data":"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"}
Jul 02 13:36:34.563206 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:34.562812    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerStarted","Data":"b4e87661505ff45aabda0405caf3e3111d7a5d020c33151cffc187839d0ccb28"}
Jul 02 13:36:38.585159 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:38.585124    2552 generic.go:356] "Generic (PLEG): container finished" podID="9015751b-c0ff-400e-b866-8774befb8951" containerID="d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8" exitCode=0
Jul 02 13:36:38.585159 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:38.585161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerDied","Data":"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"}
Jul 02 13:36:39.592037 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:39.591994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerStarted","Data":"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"}
Jul 02 13:36:39.592533 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:39.592076    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerStarted","Data":"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"}
Jul 02 13:36:39.592533 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:39.592320    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:39.592533 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:39.592374    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:36:39.614127 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:39.614069    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" podStartSLOduration=1.9731104670000001 podStartE2EDuration="6.614050867s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.6409404s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:36:39.612004508 +0000 UTC m=+1311.467397417" watchObservedRunningTime="2026-07-02 13:36:39.614050867 +0000 UTC m=+1311.469443750"
Jul 02 13:36:45.602217 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:36:45.602191    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:38:09.567002 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:09.566961    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:38:13.431467 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:13.431434    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:38:13.431893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:13.431659    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kserve-container" containerID="cri-o://61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a" gracePeriod=30
Jul 02 13:38:13.431893 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:13.431698    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kube-rbac-proxy" containerID="cri-o://1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6" gracePeriod=30
Jul 02 13:38:14.002267 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.002231    2552 generic.go:356] "Generic (PLEG): container finished" podID="9015751b-c0ff-400e-b866-8774befb8951" containerID="1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6" exitCode=2
Jul 02 13:38:14.002474 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.002308    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerDied","Data":"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"}
Jul 02 13:38:14.470934 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.470910    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:38:14.561238 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561129    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls\") pod \"9015751b-c0ff-400e-b866-8774befb8951\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") "
Jul 02 13:38:14.561238 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561172    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert\") pod \"9015751b-c0ff-400e-b866-8774befb8951\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") "
Jul 02 13:38:14.561480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561258    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location\") pod \"9015751b-c0ff-400e-b866-8774befb8951\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") "
Jul 02 13:38:14.561480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561283    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config\") pod \"9015751b-c0ff-400e-b866-8774befb8951\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") "
Jul 02 13:38:14.561480 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561347    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l6tn\" (UniqueName: \"kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn\") pod \"9015751b-c0ff-400e-b866-8774befb8951\" (UID: \"9015751b-c0ff-400e-b866-8774befb8951\") "
Jul 02 13:38:14.561620 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561569    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9015751b-c0ff-400e-b866-8774befb8951" (UID: "9015751b-c0ff-400e-b866-8774befb8951"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:38:14.561667 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.561651    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9015751b-c0ff-400e-b866-8774befb8951" (UID: "9015751b-c0ff-400e-b866-8774befb8951"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:38:14.563271 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.563234    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9015751b-c0ff-400e-b866-8774befb8951" (UID: "9015751b-c0ff-400e-b866-8774befb8951"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:38:14.563378 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.563352    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn" (OuterVolumeSpecName: "kube-api-access-9l6tn") pod "9015751b-c0ff-400e-b866-8774befb8951" (UID: "9015751b-c0ff-400e-b866-8774befb8951"). InnerVolumeSpecName "kube-api-access-9l6tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:38:14.566214 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.566189    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9015751b-c0ff-400e-b866-8774befb8951" (UID: "9015751b-c0ff-400e-b866-8774befb8951"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:38:14.661973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.661937    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015751b-c0ff-400e-b866-8774befb8951-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:14.661973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.661963    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:14.661973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.661974    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l6tn\" (UniqueName: \"kubernetes.io/projected/9015751b-c0ff-400e-b866-8774befb8951-kube-api-access-9l6tn\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:14.661973 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.661983    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015751b-c0ff-400e-b866-8774befb8951-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:14.662321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:14.661992    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9015751b-c0ff-400e-b866-8774befb8951-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:15.008155 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.008122    2552 generic.go:356] "Generic (PLEG): container finished" podID="9015751b-c0ff-400e-b866-8774befb8951" containerID="61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a" exitCode=0
Jul 02 13:38:15.008311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.008187    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"
Jul 02 13:38:15.008311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.008205    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerDied","Data":"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"}
Jul 02 13:38:15.008311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.008273    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858" event={"ID":"9015751b-c0ff-400e-b866-8774befb8951","Type":"ContainerDied","Data":"b4e87661505ff45aabda0405caf3e3111d7a5d020c33151cffc187839d0ccb28"}
Jul 02 13:38:15.008311 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.008301    2552 scope.go:117] "RemoveContainer" containerID="1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"
Jul 02 13:38:15.017086 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.017063    2552 scope.go:117] "RemoveContainer" containerID="61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"
Jul 02 13:38:15.024302 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.024285    2552 scope.go:117] "RemoveContainer" containerID="d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"
Jul 02 13:38:15.030721 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.030702    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:38:15.031508 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.031496    2552 scope.go:117] "RemoveContainer" containerID="1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"
Jul 02 13:38:15.031821 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:15.031792    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6\": container with ID starting with 1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6 not found: ID does not exist" containerID="1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"
Jul 02 13:38:15.031920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.031832    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6"} err="failed to get container status \"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6\": rpc error: code = NotFound desc = could not find container \"1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6\": container with ID starting with 1b31a6c68e295358134490069516a9c5b054b73352a26a25f6478571035f25f6 not found: ID does not exist"
Jul 02 13:38:15.031920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.031862    2552 scope.go:117] "RemoveContainer" containerID="61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"
Jul 02 13:38:15.032329 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:15.032205    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a\": container with ID starting with 61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a not found: ID does not exist" containerID="61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"
Jul 02 13:38:15.032329 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.032237    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a"} err="failed to get container status \"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a\": rpc error: code = NotFound desc = could not find container \"61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a\": container with ID starting with 61b48d138ebb6b9f77c6473103db67fe9d620e973fcaa37eef73e4af6d60d05a not found: ID does not exist"
Jul 02 13:38:15.032329 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.032258    2552 scope.go:117] "RemoveContainer" containerID="d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"
Jul 02 13:38:15.032672 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:15.032604    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8\": container with ID starting with d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8 not found: ID does not exist" containerID="d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"
Jul 02 13:38:15.032761 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.032681    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8"} err="failed to get container status \"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8\": rpc error: code = NotFound desc = could not find container \"d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8\": container with ID starting with d194aa3b760c725bf9d5d16dc1cf36c862884380c5c5e6aa5334908266b8eee8 not found: ID does not exist"
Jul 02 13:38:15.034297 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.034279    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-t2858"]
Jul 02 13:38:15.565731 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:15.565703    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9015751b-c0ff-400e-b866-8774befb8951" path="/var/lib/kubelet/pods/9015751b-c0ff-400e-b866-8774befb8951/volumes"
Jul 02 13:38:18.586855 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.586823    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:18.587216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587190    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kube-rbac-proxy"
Jul 02 13:38:18.587216 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587202    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kube-rbac-proxy"
Jul 02 13:38:18.587291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587219    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kserve-container"
Jul 02 13:38:18.587291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587224    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kserve-container"
Jul 02 13:38:18.587291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587237    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="storage-initializer"
Jul 02 13:38:18.587291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587243    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="storage-initializer"
Jul 02 13:38:18.587291 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587290    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kserve-container"
Jul 02 13:38:18.587463 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.587299    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9015751b-c0ff-400e-b866-8774befb8951" containerName="kube-rbac-proxy"
Jul 02 13:38:18.590372 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.590354    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.592988 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.592967    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:38:18.593113 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.593051    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 02 13:38:18.593113 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.593082    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f7h2p\""
Jul 02 13:38:18.593269 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.593251    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:38:18.593779 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.593762    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 02 13:38:18.593859 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.593764    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:38:18.594054 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.594040    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:38:18.599788 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.599769    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:18.693581 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.693554    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.693708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.693585    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.693708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.693619    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lfx\" (UniqueName: \"kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.693708 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.693686    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.693823 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.693714    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.794585 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.794557    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lfx\" (UniqueName: \"kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.794712 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.794623    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.794712 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.794663    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.794712 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.794699    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.794863 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.794725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.795094 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.795075    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.795308 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.795289    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.795354 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.795315    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.796937 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.796921    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.802613 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.802590    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lfx\" (UniqueName: \"kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:18.901609 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:18.901552    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:19.030770 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:19.030748    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:19.033170 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:38:19.033138    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6ebab3_61f5_4d0c_ae7b_37a0c224f9e9.slice/crio-0c7c83d7566d987eb7b2a24d1cfff73dc69303b1225243e0155c69aa0c33e54b WatchSource:0}: Error finding container 0c7c83d7566d987eb7b2a24d1cfff73dc69303b1225243e0155c69aa0c33e54b: Status 404 returned error can't find the container with id 0c7c83d7566d987eb7b2a24d1cfff73dc69303b1225243e0155c69aa0c33e54b
Jul 02 13:38:20.031020 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:20.030977    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerStarted","Data":"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"}
Jul 02 13:38:20.031380 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:20.031056    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerStarted","Data":"0c7c83d7566d987eb7b2a24d1cfff73dc69303b1225243e0155c69aa0c33e54b"}
Jul 02 13:38:25.054045 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:25.054018    2552 generic.go:356] "Generic (PLEG): container finished" podID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerID="92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6" exitCode=0
Jul 02 13:38:25.054355 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:25.054087    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerDied","Data":"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"}
Jul 02 13:38:26.061473 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.061440    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerStarted","Data":"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"}
Jul 02 13:38:26.061868 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.061499    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerStarted","Data":"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"}
Jul 02 13:38:26.061868 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.061757    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:26.061868 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.061806    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:26.063156 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.063131    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:38:26.086830 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:26.086784    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podStartSLOduration=2.21262727 podStartE2EDuration="8.086751827s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.874124557s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:38:26.086395791 +0000 UTC m=+1417.941788712" watchObservedRunningTime="2026-07-02 13:38:26.086751827 +0000 UTC m=+1417.942144711"
Jul 02 13:38:27.066483 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:27.066439    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:38:32.071324 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:32.071295    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:32.071811 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:32.071781    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:38:42.072059 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:42.071985    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:48.640505 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:48.640472    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:48.640978 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:48.640781    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container" containerID="cri-o://882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c" gracePeriod=30
Jul 02 13:38:48.640978 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:48.640889    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kube-rbac-proxy" containerID="cri-o://99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4" gracePeriod=30
Jul 02 13:38:49.172094 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.172041    2552 generic.go:356] "Generic (PLEG): container finished" podID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerID="99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4" exitCode=2
Jul 02 13:38:49.172266 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.172112    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerDied","Data":"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"}
Jul 02 13:38:49.376398 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.376377    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:49.444932 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.444846    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location\") pod \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") "
Jul 02 13:38:49.444932 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.444900    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert\") pod \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") "
Jul 02 13:38:49.445164 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.444947    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls\") pod \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") "
Jul 02 13:38:49.445164 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445019    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8lfx\" (UniqueName: \"kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx\") pod \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") "
Jul 02 13:38:49.445164 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445047    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config\") pod \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\" (UID: \"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9\") "
Jul 02 13:38:49.445322 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445242    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" (UID: "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:38:49.445384 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445320    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-cabundle-cert\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:49.445461 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445434    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" (UID: "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:38:49.445923 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.445905    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" (UID: "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:38:49.447038 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.447004    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx" (OuterVolumeSpecName: "kube-api-access-k8lfx") pod "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" (UID: "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9"). InnerVolumeSpecName "kube-api-access-k8lfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:38:49.447136 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.447038    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" (UID: "6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:38:49.545824 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.545803    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kserve-provision-location\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:49.545824 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.545823    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-proxy-tls\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:49.545977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.545834    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lfx\" (UniqueName: \"kubernetes.io/projected/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-api-access-k8lfx\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:49.545977 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:49.545845    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:38:50.178711 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.178678    2552 generic.go:356] "Generic (PLEG): container finished" podID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerID="882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c" exitCode=0
Jul 02 13:38:50.179105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.178735    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"
Jul 02 13:38:50.179105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.178760    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerDied","Data":"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"}
Jul 02 13:38:50.179105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.178826    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf" event={"ID":"6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9","Type":"ContainerDied","Data":"0c7c83d7566d987eb7b2a24d1cfff73dc69303b1225243e0155c69aa0c33e54b"}
Jul 02 13:38:50.179105 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.178864    2552 scope.go:117] "RemoveContainer" containerID="99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"
Jul 02 13:38:50.187035 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.187020    2552 scope.go:117] "RemoveContainer" containerID="882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"
Jul 02 13:38:50.194185 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.194168    2552 scope.go:117] "RemoveContainer" containerID="92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"
Jul 02 13:38:50.198700 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.198678    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:50.201986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.201966    2552 scope.go:117] "RemoveContainer" containerID="99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"
Jul 02 13:38:50.202196 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202180    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xhdwf"]
Jul 02 13:38:50.202196 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:50.202186    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4\": container with ID starting with 99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4 not found: ID does not exist" containerID="99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"
Jul 02 13:38:50.202284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202209    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4"} err="failed to get container status \"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4\": rpc error: code = NotFound desc = could not find container \"99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4\": container with ID starting with 99a08dab950c14fa12bbd2fa2a218763b3c094f178b6011c9f7cf2be0c4657e4 not found: ID does not exist"
Jul 02 13:38:50.202284 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202224    2552 scope.go:117] "RemoveContainer" containerID="882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"
Jul 02 13:38:50.202472 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:50.202453    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c\": container with ID starting with 882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c not found: ID does not exist" containerID="882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"
Jul 02 13:38:50.202536 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202482    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c"} err="failed to get container status \"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c\": rpc error: code = NotFound desc = could not find container \"882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c\": container with ID starting with 882183fe5f243de2932651e530e770127c391e0ed201b58a941545ba6351af7c not found: ID does not exist"
Jul 02 13:38:50.202536 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202506    2552 scope.go:117] "RemoveContainer" containerID="92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"
Jul 02 13:38:50.202705 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:38:50.202691    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6\": container with ID starting with 92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6 not found: ID does not exist" containerID="92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"
Jul 02 13:38:50.202757 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:50.202711    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6"} err="failed to get container status \"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6\": rpc error: code = NotFound desc = could not find container \"92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6\": container with ID starting with 92f64d4d8565fb65797f808daf32a9f4f04a47306d9182f2a7a4fb1588c6f8a6 not found: ID does not exist"
Jul 02 13:38:51.565257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:51.565224    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" path="/var/lib/kubelet/pods/6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9/volumes"
Jul 02 13:38:55.237592 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237559    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5t786/must-gather-nrr9t"]
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237917    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kube-rbac-proxy"
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237929    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kube-rbac-proxy"
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237937    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="storage-initializer"
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237942    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="storage-initializer"
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237954    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container"
Jul 02 13:38:55.237976 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.237960    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container"
Jul 02 13:38:55.238213 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.238013    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kube-rbac-proxy"
Jul 02 13:38:55.238213 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.238024    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6ebab3-61f5-4d0c-ae7b-37a0c224f9e9" containerName="kserve-container"
Jul 02 13:38:55.242609 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.242592    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.245680 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.245658    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5t786\"/\"default-dockercfg-zgmkb\""
Jul 02 13:38:55.246425 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.246402    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5t786\"/\"kube-root-ca.crt\""
Jul 02 13:38:55.246562 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.246546    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5t786\"/\"openshift-service-ca.crt\""
Jul 02 13:38:55.269948 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.269926    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5t786/must-gather-nrr9t"]
Jul 02 13:38:55.388913 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.388866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj5nx\" (UniqueName: \"kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.389033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.388979    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.490150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.490076    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.490150 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.490130    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj5nx\" (UniqueName: \"kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.490436 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.490414    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.498992 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.498970    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj5nx\" (UniqueName: \"kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx\") pod \"must-gather-nrr9t\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") " pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.569098 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.569077    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:38:55.695406 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.695384    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5t786/must-gather-nrr9t"]
Jul 02 13:38:55.697846 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:38:55.697822    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod217cf491_0a03_4c35_b35f_ad1ec11c77ca.slice/crio-06bcfe1755d8a97a03208126475b030d8988727ad8042f92f1e52416d3d4062e WatchSource:0}: Error finding container 06bcfe1755d8a97a03208126475b030d8988727ad8042f92f1e52416d3d4062e: Status 404 returned error can't find the container with id 06bcfe1755d8a97a03208126475b030d8988727ad8042f92f1e52416d3d4062e
Jul 02 13:38:55.699993 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:55.699978    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:38:56.205343 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:38:56.205313    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5t786/must-gather-nrr9t" event={"ID":"217cf491-0a03-4c35-b35f-ad1ec11c77ca","Type":"ContainerStarted","Data":"06bcfe1755d8a97a03208126475b030d8988727ad8042f92f1e52416d3d4062e"}
Jul 02 13:39:02.238474 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:02.238432    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5t786/must-gather-nrr9t" event={"ID":"217cf491-0a03-4c35-b35f-ad1ec11c77ca","Type":"ContainerStarted","Data":"cdc6b59a9865d85030eafb9ad4947a36d25d657093e9a963939ba68dfbea53e8"}
Jul 02 13:39:02.238918 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:02.238486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5t786/must-gather-nrr9t" event={"ID":"217cf491-0a03-4c35-b35f-ad1ec11c77ca","Type":"ContainerStarted","Data":"8ca501cb15af9a7eed24b39a4480947cb3e28f51a518cfbd41fb9514273bbbc9"}
Jul 02 13:39:02.257485 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:02.257437    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5t786/must-gather-nrr9t" podStartSLOduration=1.256537842 podStartE2EDuration="7.257422926s" totalImagesPullingTime="6.000885084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:02.255198629 +0000 UTC m=+1454.110591519" watchObservedRunningTime="2026-07-02 13:39:02.257422926 +0000 UTC m=+1454.112815810"
Jul 02 13:39:22.334794 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:22.334760    2552 generic.go:356] "Generic (PLEG): container finished" podID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerID="8ca501cb15af9a7eed24b39a4480947cb3e28f51a518cfbd41fb9514273bbbc9" exitCode=0
Jul 02 13:39:22.335197 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:22.334831    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5t786/must-gather-nrr9t" event={"ID":"217cf491-0a03-4c35-b35f-ad1ec11c77ca","Type":"ContainerDied","Data":"8ca501cb15af9a7eed24b39a4480947cb3e28f51a518cfbd41fb9514273bbbc9"}
Jul 02 13:39:22.335197 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:22.335141    2552 scope.go:117] "RemoveContainer" containerID="8ca501cb15af9a7eed24b39a4480947cb3e28f51a518cfbd41fb9514273bbbc9"
Jul 02 13:39:22.728774 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:22.728753    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5t786_must-gather-nrr9t_217cf491-0a03-4c35-b35f-ad1ec11c77ca/gather/0.log"
Jul 02 13:39:26.012418 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:26.012386    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-stzdv_3f6ceabc-fa61-470e-8df7-db311b0433f6/global-pull-secret-syncer/0.log"
Jul 02 13:39:26.239327 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:26.239303    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tvlpw_354e948f-dc63-416a-8642-8598553c6185/konnectivity-agent/0.log"
Jul 02 13:39:26.362552 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:26.362488    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-149.ec2.internal_d0695f0d6e8402fe098c22a36367da2e/haproxy/0.log"
Jul 02 13:39:28.137055 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.137021    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5t786/must-gather-nrr9t"]
Jul 02 13:39:28.137531 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.137202    2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-5t786/must-gather-nrr9t" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="copy" containerID="cri-o://cdc6b59a9865d85030eafb9ad4947a36d25d657093e9a963939ba68dfbea53e8" gracePeriod=2
Jul 02 13:39:28.142587 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.142561    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5t786/must-gather-nrr9t"]
Jul 02 13:39:28.368593 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.368566    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5t786_must-gather-nrr9t_217cf491-0a03-4c35-b35f-ad1ec11c77ca/copy/0.log"
Jul 02 13:39:28.368972 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.368947    2552 generic.go:356] "Generic (PLEG): container finished" podID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerID="cdc6b59a9865d85030eafb9ad4947a36d25d657093e9a963939ba68dfbea53e8" exitCode=143
Jul 02 13:39:28.369057 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.369003    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06bcfe1755d8a97a03208126475b030d8988727ad8042f92f1e52416d3d4062e"
Jul 02 13:39:28.371904 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.371887    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5t786_must-gather-nrr9t_217cf491-0a03-4c35-b35f-ad1ec11c77ca/copy/0.log"
Jul 02 13:39:28.373393 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.373378    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:28.375328 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:39:28.375306    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrr9t\" is forbidden: User \"system:node:ip-10-0-141-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5t786\": no relationship found between node 'ip-10-0-141-149.ec2.internal' and this object" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:28.487220 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.487194    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output\") pod \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") "
Jul 02 13:39:28.487368 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.487294    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj5nx\" (UniqueName: \"kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx\") pod \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\" (UID: \"217cf491-0a03-4c35-b35f-ad1ec11c77ca\") "
Jul 02 13:39:28.488539 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.488516    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "217cf491-0a03-4c35-b35f-ad1ec11c77ca" (UID: "217cf491-0a03-4c35-b35f-ad1ec11c77ca"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:28.489376 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.489357    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx" (OuterVolumeSpecName: "kube-api-access-mj5nx") pod "217cf491-0a03-4c35-b35f-ad1ec11c77ca" (UID: "217cf491-0a03-4c35-b35f-ad1ec11c77ca"). InnerVolumeSpecName "kube-api-access-mj5nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:28.587915 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.587893    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj5nx\" (UniqueName: \"kubernetes.io/projected/217cf491-0a03-4c35-b35f-ad1ec11c77ca-kube-api-access-mj5nx\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:39:28.587915 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:28.587913    2552 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/217cf491-0a03-4c35-b35f-ad1ec11c77ca-must-gather-output\") on node \"ip-10-0-141-149.ec2.internal\" DevicePath \"\""
Jul 02 13:39:29.372947 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:29.372916    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:29.375216 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:39:29.375188    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrr9t\" is forbidden: User \"system:node:ip-10-0-141-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5t786\": no relationship found between node 'ip-10-0-141-149.ec2.internal' and this object" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:29.382376 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:39:29.382354    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrr9t\" is forbidden: User \"system:node:ip-10-0-141-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5t786\": no relationship found between node 'ip-10-0-141-149.ec2.internal' and this object" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:29.565134 ip-10-0-141-149 kubenswrapper[2552]: E0702 13:39:29.565102    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrr9t\" is forbidden: User \"system:node:ip-10-0-141-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5t786\": no relationship found between node 'ip-10-0-141-149.ec2.internal' and this object" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" pod="openshift-must-gather-5t786/must-gather-nrr9t"
Jul 02 13:39:29.565992 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:29.565973    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" path="/var/lib/kubelet/pods/217cf491-0a03-4c35-b35f-ad1ec11c77ca/volumes"
Jul 02 13:39:30.278834 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.278806    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/alertmanager/0.log"
Jul 02 13:39:30.300751 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.300729    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/config-reloader/0.log"
Jul 02 13:39:30.324664 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.324643    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/kube-rbac-proxy-web/0.log"
Jul 02 13:39:30.347829 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.347809    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/kube-rbac-proxy/0.log"
Jul 02 13:39:30.370592 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.370569    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/kube-rbac-proxy-metric/0.log"
Jul 02 13:39:30.395282 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.395264    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/prom-label-proxy/0.log"
Jul 02 13:39:30.422192 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.422173    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_53ee3f7e-3228-45c9-ac95-b4a9416914e8/init-config-reloader/0.log"
Jul 02 13:39:30.509920 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.509901    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-state-metrics/0.log"
Jul 02 13:39:30.557952 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.557886    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-rbac-proxy-main/0.log"
Jul 02 13:39:30.591488 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.591465    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-ctlsz_0a472ad0-3f79-4836-b998-eaa7ddf9f939/kube-rbac-proxy-self/0.log"
Jul 02 13:39:30.621882 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.621843    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b57f869fd-ppsvt_894ee300-2398-4436-977c-82973679ed08/metrics-server/0.log"
Jul 02 13:39:30.652411 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.652389    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-4bkrh_beefa50a-cabb-4346-9ebb-e9a12b9e5acc/monitoring-plugin/0.log"
Jul 02 13:39:30.798693 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.798666    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/node-exporter/0.log"
Jul 02 13:39:30.823983 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.823919    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/kube-rbac-proxy/0.log"
Jul 02 13:39:30.854351 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.854333    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mmmhc_7a4393e8-d4e2-469d-8d5e-b9a4ab1b964b/init-textfile/0.log"
Jul 02 13:39:30.966682 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.966660    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/kube-rbac-proxy-main/0.log"
Jul 02 13:39:30.989202 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:30.989181    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/kube-rbac-proxy-self/0.log"
Jul 02 13:39:31.011478 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.011460    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-jc98n_e7a15b67-891f-40f8-a64b-30bb3a2ce04e/openshift-state-metrics/0.log"
Jul 02 13:39:31.050256 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.050227    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/prometheus/0.log"
Jul 02 13:39:31.072633 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.072612    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/config-reloader/0.log"
Jul 02 13:39:31.097745 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.097697    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/thanos-sidecar/0.log"
Jul 02 13:39:31.122033 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.122012    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/kube-rbac-proxy-web/0.log"
Jul 02 13:39:31.145491 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.145469    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/kube-rbac-proxy/0.log"
Jul 02 13:39:31.171233 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.171205    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/kube-rbac-proxy-thanos/0.log"
Jul 02 13:39:31.196656 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.196639    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b5c3146-9b10-447b-ae11-b9c8c80a783d/init-config-reloader/0.log"
Jul 02 13:39:31.320013 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.319997    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/telemeter-client/0.log"
Jul 02 13:39:31.346229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.346214    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/reload/0.log"
Jul 02 13:39:31.377198 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.377182    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787f69c8cd-rxqmq_2f793825-c7b1-4c69-9320-07d0ddd83b0f/kube-rbac-proxy/0.log"
Jul 02 13:39:31.421260 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.421243    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/thanos-query/0.log"
Jul 02 13:39:31.453547 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.453531    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-web/0.log"
Jul 02 13:39:31.479524 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.479507    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy/0.log"
Jul 02 13:39:31.502359 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.502337    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/prom-label-proxy/0.log"
Jul 02 13:39:31.532353 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.532329    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-rules/0.log"
Jul 02 13:39:31.555474 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:31.555457    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ccd878888-872p5_282fcef4-e4e9-4b69-8edf-62c585b3ffca/kube-rbac-proxy-metrics/0.log"
Jul 02 13:39:33.992814 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:33.992785    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-869d497466-z5474_bec8005f-d24d-40ba-b0e6-41aa5f671828/console/0.log"
Jul 02 13:39:34.362134 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362066    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"]
Jul 02 13:39:34.362472 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362452    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="copy"
Jul 02 13:39:34.362472 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362472    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="copy"
Jul 02 13:39:34.362472 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362482    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="gather"
Jul 02 13:39:34.362723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362488    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="gather"
Jul 02 13:39:34.362723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362588    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="copy"
Jul 02 13:39:34.362723 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.362697    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="217cf491-0a03-4c35-b35f-ad1ec11c77ca" containerName="gather"
Jul 02 13:39:34.365329 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.365314    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.367860 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.367836    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95kqz\"/\"openshift-service-ca.crt\""
Jul 02 13:39:34.367998 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.367921    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95kqz\"/\"default-dockercfg-jsxwb\""
Jul 02 13:39:34.367998 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.367986    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95kqz\"/\"kube-root-ca.crt\""
Jul 02 13:39:34.375317 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.375295    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"]
Jul 02 13:39:34.434116 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.434089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-podres\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.434244 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.434127    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt27t\" (UniqueName: \"kubernetes.io/projected/f15d379d-2a1b-485e-8744-d571006e623b-kube-api-access-vt27t\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.434244 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.434146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-proc\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.434321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.434258    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-lib-modules\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.434321 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.434302    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-sys\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535166 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535144    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt27t\" (UniqueName: \"kubernetes.io/projected/f15d379d-2a1b-485e-8744-d571006e623b-kube-api-access-vt27t\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535277 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535171    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-proc\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535277 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-lib-modules\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535277 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535239    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-sys\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535282    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-proc\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535288    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-podres\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535329    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-sys\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535382    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-podres\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.535419 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.535397    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15d379d-2a1b-485e-8744-d571006e623b-lib-modules\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.543692 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.543670    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt27t\" (UniqueName: \"kubernetes.io/projected/f15d379d-2a1b-485e-8744-d571006e623b-kube-api-access-vt27t\") pod \"perf-node-gather-daemonset-4qzg9\" (UID: \"f15d379d-2a1b-485e-8744-d571006e623b\") " pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.675540 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.675480    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:34.801257 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:34.801230    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"]
Jul 02 13:39:34.804792 ip-10-0-141-149 kubenswrapper[2552]: W0702 13:39:34.804768    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf15d379d_2a1b_485e_8744_d571006e623b.slice/crio-e52d32805ca17404e1f8f099e271bffc5863d35e6bbbdede59aaca632bf5dcb0 WatchSource:0}: Error finding container e52d32805ca17404e1f8f099e271bffc5863d35e6bbbdede59aaca632bf5dcb0: Status 404 returned error can't find the container with id e52d32805ca17404e1f8f099e271bffc5863d35e6bbbdede59aaca632bf5dcb0
Jul 02 13:39:35.398890 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.398843    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9" event={"ID":"f15d379d-2a1b-485e-8744-d571006e623b","Type":"ContainerStarted","Data":"b9c900b6c0d0ac67642dbec5dcfb3671e4472f776c55b6c04c19caf2e337e39b"}
Jul 02 13:39:35.399237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.398910    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9" event={"ID":"f15d379d-2a1b-485e-8744-d571006e623b","Type":"ContainerStarted","Data":"e52d32805ca17404e1f8f099e271bffc5863d35e6bbbdede59aaca632bf5dcb0"}
Jul 02 13:39:35.399237 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.398937    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:35.417381 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.417322    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9" podStartSLOduration=1.417307852 podStartE2EDuration="1.417307852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:35.416159846 +0000 UTC m=+1487.271552754" watchObservedRunningTime="2026-07-02 13:39:35.417307852 +0000 UTC m=+1487.272700736"
Jul 02 13:39:35.478014 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.477986    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-92cmb_ee0f85d9-e141-4201-b5fc-c1bc593f6fc7/dns/0.log"
Jul 02 13:39:35.501741 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.501717    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-92cmb_ee0f85d9-e141-4201-b5fc-c1bc593f6fc7/kube-rbac-proxy/0.log"
Jul 02 13:39:35.632307 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:35.632287    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-546hs_1cb9273a-103d-4f41-b823-1a2c4f6029d5/dns-node-resolver/0.log"
Jul 02 13:39:36.259264 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:36.259240    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r8wzf_a04c8f06-90e1-4c77-834b-e9d44cf8bfac/node-ca/0.log"
Jul 02 13:39:37.421229 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:37.421187    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w57dk_720c4f68-b38c-4924-9d7f-f027bf16cdd8/serve-healthcheck-canary/0.log"
Jul 02 13:39:37.848633 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:37.848607    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-26spv_3cdca863-8d86-4583-8838-4607100cc261/kube-rbac-proxy/0.log"
Jul 02 13:39:37.869404 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:37.869381    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-26spv_3cdca863-8d86-4583-8838-4607100cc261/exporter/0.log"
Jul 02 13:39:37.892607 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:37.892589    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-26spv_3cdca863-8d86-4583-8838-4607100cc261/extractor/0.log"
Jul 02 13:39:39.996992 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:39.996959    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-694f5b7594-v2s6v_db2df7fa-b894-45f2-8d73-b4a88d2d789b/manager/0.log"
Jul 02 13:39:40.106375 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.106345    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-2scph_a286c0f3-5d8b-4cd0-a7ec-6b3a8b511541/manager/0.log"
Jul 02 13:39:40.136276 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.136257    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pt9rk_dedef70c-57a2-450c-8bc0-45c5e2443ce2/s3-init/0.log"
Jul 02 13:39:40.151006 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.150986    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pt9rk_dedef70c-57a2-450c-8bc0-45c5e2443ce2/download-hf-model/0.log"
Jul 02 13:39:40.177371 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.177352    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6hq7f_264811d2-d4ff-4f91-af26-b0aea8898068/s3-tls-init-custom/0.log"
Jul 02 13:39:40.192775 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.192756    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6hq7f_264811d2-d4ff-4f91-af26-b0aea8898068/download-hf-model/0.log"
Jul 02 13:39:40.219165 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.219148    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-82h99_c4af3453-caf5-4441-aeb2-cd3103bf8a75/s3-tls-init-serving/0.log"
Jul 02 13:39:40.236563 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:40.236539    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-82h99_c4af3453-caf5-4441-aeb2-cd3103bf8a75/download-hf-model/0.log"
Jul 02 13:39:41.413716 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:41.413688    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95kqz/perf-node-gather-daemonset-4qzg9"
Jul 02 13:39:45.696437 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.696412    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/kube-multus-additional-cni-plugins/0.log"
Jul 02 13:39:45.719649 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.719627    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/egress-router-binary-copy/0.log"
Jul 02 13:39:45.745732 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.745709    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/cni-plugins/0.log"
Jul 02 13:39:45.771373 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.771357    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/bond-cni-plugin/0.log"
Jul 02 13:39:45.799405 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.799386    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/routeoverride-cni/0.log"
Jul 02 13:39:45.821670 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.821653    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/whereabouts-cni-bincopy/0.log"
Jul 02 13:39:45.847024 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:45.847006    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s89wm_a62feb9f-c762-41b5-804b-7abe8c0facac/whereabouts-cni/0.log"
Jul 02 13:39:46.297706 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:46.297685    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rlnbl_5b1fe7e5-d0c5-42fa-aec8-691835f4ae2e/kube-multus/0.log"
Jul 02 13:39:46.370944 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:46.370925    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q9mnj_c86e3245-2705-4a72-b608-61bebd9eb17e/network-metrics-daemon/0.log"
Jul 02 13:39:46.403579 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:46.403556    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q9mnj_c86e3245-2705-4a72-b608-61bebd9eb17e/kube-rbac-proxy/0.log"
Jul 02 13:39:47.500591 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.500560    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/ovn-controller/0.log"
Jul 02 13:39:47.523705 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.523679    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/ovn-acl-logging/0.log"
Jul 02 13:39:47.547381 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.547358    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/kube-rbac-proxy-node/0.log"
Jul 02 13:39:47.567749 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.567726    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 13:39:47.590643 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.590626    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/northd/0.log"
Jul 02 13:39:47.619617 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.619597    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/nbdb/0.log"
Jul 02 13:39:47.677594 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.677512    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/sbdb/0.log"
Jul 02 13:39:47.742141 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:47.742119    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nkxrg_e636dd3e-e640-47e6-8d2f-b16930340638/ovnkube-controller/0.log"
Jul 02 13:39:48.644248 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:48.644216    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5t786_must-gather-nrr9t_217cf491-0a03-4c35-b35f-ad1ec11c77ca/copy/0.log"
Jul 02 13:39:48.648462 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:48.648438    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5t786_must-gather-nrr9t_217cf491-0a03-4c35-b35f-ad1ec11c77ca/copy/0.log"
Jul 02 13:39:49.065615 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:49.065593    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w5957_6354334f-1524-4060-948b-c568d9382894/network-check-target-container/0.log"
Jul 02 13:39:50.055515 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:50.055491    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g8qvd_37e92131-f21d-4309-9f51-9215d2221123/iptables-alerter/0.log"
Jul 02 13:39:50.740986 ip-10-0-141-149 kubenswrapper[2552]: I0702 13:39:50.740960    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-gmjck_3590789f-cff0-43ff-8158-e0f5ab08ea49/tuned/0.log"