Jul 27 21:03:54.839813 ip-10-0-135-121 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:03:55.328459 ip-10-0-135-121 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:55.328459 ip-10-0-135-121 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:03:55.328459 ip-10-0-135-121 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:55.328459 ip-10-0-135-121 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:03:55.328459 ip-10-0-135-121 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:55.330685 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.330577    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:03:55.337465 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337451    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.337465 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337465    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337470    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337475    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337480    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337484    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337487    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337500    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337503    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337506    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337509    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337512    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337514    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337517    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337520    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337522    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337528    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337531    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337534    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337537    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.337537 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337541    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337545    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337549    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337552    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337554    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337557    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337559    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337562    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337564    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337567    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337569    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337572    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337574    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337577    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337579    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337582    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337585    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337587    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337590    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.337994 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337592    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337595    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337597    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337601    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337604    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337607    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337611    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337613    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337616    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337618    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337620    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337623    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337626    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337629    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337632    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337634    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337637    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337639    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337642    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.338473 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337644    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337648    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337650    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337653    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337655    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337658    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337661    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337664    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337666    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337668    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337671    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337673    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337676    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337678    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337681    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337684    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337686    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337689    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337692    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337695    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.338978 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337698    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337700    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337702    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337705    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337708    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337710    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337713    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337715    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337718    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337721    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337723    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337726    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337728    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337731    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337733    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337736    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337738    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337741    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337743    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337746    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.339448 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.337748    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338440    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338447    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338450    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338453    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338456    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338458    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338461    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338464    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338466    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338470    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338475    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338478    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338482    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338485    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338503    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338506    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338510    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338512    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338515    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.339934 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338518    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338520    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338523    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338526    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338529    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338531    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338534    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338536    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338539    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338542    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338544    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338547    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338550    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338553    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338556    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338558    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338560    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338563    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338565    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.340415 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338568    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338570    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338573    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338576    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338579    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338581    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338584    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338587    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338590    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338593    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338596    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338600    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338603    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338606    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338609    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338612    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338615    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338617    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338620    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.340879 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338624    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338627    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338629    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338632    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338635    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338637    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338640    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338643    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338645    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338648    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338650    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338653    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338655    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338658    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338660    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338663    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338665    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338668    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338670    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338673    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.341357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338676    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338679    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338682    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338685    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338687    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338690    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338693    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338695    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338698    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338700    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338702    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338705    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338707    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338710    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338712    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338714    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338717    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338719    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338721    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338724    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.341840 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338727    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.338729    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339079    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339084    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339087    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339090    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339093    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339097    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339100    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339102    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339105    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339108    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339111    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339113    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339116    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339119    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339122    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339124    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339127    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.342331 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339130    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339132    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339134    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339137    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339140    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339142    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339145    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339147    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339150    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339152    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339155    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339157    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339160    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339163    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339166    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339168    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339171    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339174    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339176    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339179    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.342794 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339181    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339183    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339186    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339188    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339191    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339193    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339196    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339198    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339201    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339204    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339207    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339209    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339212    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339214    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339217    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339220    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339223    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339226    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339228    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339231    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.343270 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339233    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339236    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339238    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339241    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339244    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339246    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339249    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339252    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339254    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339257    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339259    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339262    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339265    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339267    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339270    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339272    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339275    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339277    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339281    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.343785 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339284    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339287    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339290    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339293    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339296    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339299    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339301    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339304    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339306    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339309    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339314    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339317    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339319    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339321    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339324    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339326    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339329    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339331    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339333    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339336    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.344238 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339339    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339341    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.339345    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339419    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339425    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339431    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339435    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339439    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339442    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339446    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339450    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339454    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339457    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339460    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339464    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339467    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339469    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339472    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339476    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339479    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339482    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339485    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339502    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339506    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:03:55.344730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339509    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339512    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339516    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339519    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339522    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339526    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339529    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339532    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339535    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339538    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339542    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339546    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339549    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339552    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339554    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339558    2555 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339561    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339565    2555 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339568    2555 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339571    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339574    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339577    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339580    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339583    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339586    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:03:55.345311 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339588    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339591    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339594    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339597    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339600    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339604    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339607    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339610    2555 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339613    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339616    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339619    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339622    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339625    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339628    2555 flags.go:64] FLAG: --help="false"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339630    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-135-121.ec2.internal"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339633    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339636    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339640    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339644    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339647    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339650    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339653    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339656    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:03:55.345913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339659    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339662    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339665    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339667    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339670    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339673    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339676    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339678    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339681    2555 flags.go:64] FLAG: --lock-file=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339684    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339687    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339690    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339695    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339697    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339701    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339704    2555 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339707    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339710    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339713    2555 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339716    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339720    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339723    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339727    2555 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339730    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339733    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:03:55.346485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339736    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339739    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339742    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339745    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339748    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339755    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339758    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339761    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339764    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339767    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339772    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339775    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339778    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339781    2555 flags.go:64] FLAG: --port="10250"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339784    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339787    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c98f3b8d814ab5d3"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339790    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339792    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339795    2555 flags.go:64] FLAG: --register-node="true"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339798    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339803    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339806    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339809    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339812    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339815    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:03:55.347140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339818    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339822    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339825    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339828    2555 flags.go:64] FLAG: --runonce="false"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339830    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339834    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339837    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339840    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339842    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339845    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339849    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339852    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339855    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339858    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339861    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339863    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339867    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339869    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339872    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339877    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339880    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339882    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339885    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339888    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339890    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:03:55.347766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339893    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339896    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339899    2555 flags.go:64] FLAG: --v="2"
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339904    2555 flags.go:64] FLAG: --version="false"
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339907    2555 flags.go:64] FLAG: --vmodule=""
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339911    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.339914    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340671    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340678    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340681    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340684    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340687    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340691    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340694    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340696    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340699    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340701    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340704    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340707    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340710    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340712    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.348396 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340715    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340718    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340720    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340722    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340725    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340727    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340730    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340732    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340735    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340737    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340741    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340744    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340747    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340750    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340753    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340756    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340759    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340762    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340764    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.348911 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340767    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340769    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340771    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340774    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340778    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340781    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340783    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340786    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340788    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340791    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340793    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340796    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340799    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340801    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340804    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340806    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340809    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340811    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340814    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340816    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.349357 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340819    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340821    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340824    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340826    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340828    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340831    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340833    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340836    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340839    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340841    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340844    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340846    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340848    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340851    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340854    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340856    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340860    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340863    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340865    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340868    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.349848 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340870    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340873    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340875    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340878    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340881    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340884    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340886    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340889    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340891    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340894    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340897    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340899    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340902    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340904    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340907    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340909    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340912    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340914    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340917    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340920    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.350332 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340923    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340926    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340929    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340931    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340934    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.340936    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.340941    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349110    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349124    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349142    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349148    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349218    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349223    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349226    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.350867 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349229    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349232    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349240    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349243    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349246    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349248    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349251    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349254    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349256    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349258    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349261    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349264    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349266    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349269    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349271    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349273    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349276    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349279    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349282    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349284    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.351206 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349286    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349289    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349291    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349294    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349296    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349299    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349302    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349305    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349307    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349310    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349312    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349315    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349317    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349320    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349323    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349331    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349339    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349342    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349344    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349346    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.351783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349349    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349351    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349354    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349357    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349359    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349362    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349364    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349367    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349369    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349372    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349374    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349377    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349379    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349382    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349384    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349387    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349389    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349392    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349395    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.352567 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349399    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349404    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349408    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349411    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349414    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349417    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349420    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349423    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349425    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349433    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349436    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349439    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349441    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349444    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349446    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349449    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349451    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349454    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349456    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349458    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.353130 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349461    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349464    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349468    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349471    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349474    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349478    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349480    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349483    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349486    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349502    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349506    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349510    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349513    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349516    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349518    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349521    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.353667 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349523    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349528    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349680    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349685    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349688    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349690    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349699    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349702    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349705    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349707    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349710    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349713    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349715    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349718    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349720    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:55.354049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349723    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349725    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349728    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349730    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349733    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349735    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349738    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349740    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349743    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349745    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349747    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349750    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349752    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349755    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349758    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349761    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349764    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349767    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349769    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:55.354406 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349772    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349774    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349776    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349779    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349781    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349790    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349793    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349795    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349798    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349801    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349805    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349808    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349811    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349813    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349816    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349819    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349821    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349824    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349826    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:55.354860 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349829    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349831    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349834    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349836    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349839    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349841    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349844    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349846    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349849    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349851    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349854    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349857    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349867    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349870    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349873    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349875    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349878    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349880    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349882    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349891    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:55.355300 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349893    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349896    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349898    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349900    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349903    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349905    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349907    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349910    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349912    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349915    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349918    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349920    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349923    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349925    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349927    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349930    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349932    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349935    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349937    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:55.355777 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349939    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349942    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349944    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349947    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349949    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349951    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349954    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349956    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:55.349958    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.349963    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.350066    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.354935    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:03:55.356209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.356182    2555 server.go:1019] "Starting client certificate rotation"
Jul 27 21:03:55.356502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.356274    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:03:55.356502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.356313    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:03:55.359333 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.359319    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:03:55.361179 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.361163    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:03:55.373431 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.373412    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:03:55.375918 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.375903    2555 log.go:25] "Validated CRI v1 image API"
Jul 27 21:03:55.376736 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.376715    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:03:55.379055 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.379032    2555 fs.go:135] Filesystem UUIDs: map[3699ab14-5ca7-49dc-adf9-bcb8b6c9a00e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cf4e3f8f-4a09-460d-a842-3bc4bf748abd:/dev/nvme0n1p4]
Jul 27 21:03:55.379117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:55.379055    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:03:56.678584 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.678462    2555 manager.go:216] Machine: {Timestamp:2026-07-27 21:03:56.677351745 +0000 UTC m=+0.347843383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199679 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27104033ed0e71697e39b07a1c4a59 SystemUUID:ec271040-33ed-0e71-697e-39b07a1c4a59 BootID:fffd0855-bab5-4b7c-8d53-0feca9971914 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:24:ce:13:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:24:ce:13:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:ea:41:62:d6:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:03:56.678584 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.678560    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:03:56.678737 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.678628    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:03:56.682210 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.682188    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:03:56.682348 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.682214    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:03:56.683440 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.683425    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:03:56.684126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.684108    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:03:56.684164 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.684128    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:03:56.684164 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.684136    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:03:56.685665 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.685653    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:03:56.687722 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.687712    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:03:56.692695 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692683    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:03:56.692828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692818    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:03:56.692871 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692847    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:03:56.692871 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692862    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:03:56.692955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692875    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:03:56.692955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692890    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:03:56.692955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.692921    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:03:56.693561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.693537    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:03:56.693652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.693622    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:03:56.695593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.695572    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:03:56.698762 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.698738    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:03:56.698825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.698782    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:03:56.701491 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701478    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701495    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701502    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701508    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701516    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701521    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701529    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701534    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:03:56.701536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701539    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:03:56.701744 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701546    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:03:56.701744 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701554    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:03:56.701744 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.701563    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:03:56.702175 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.702164    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:03:56.702209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.702176    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:03:56.704526 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.704498    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:03:56.704526 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.704492    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:03:56.704643 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.704576    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:03:56.705313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.705299    2555 server.go:1268] "Started kubelet"
Jul 27 21:03:56.705420 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.705389    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:03:56.705790 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.705473    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:03:56.705790 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.705521    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:03:56.705790 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.705548    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ljzk4"
Jul 27 21:03:56.706159 ip-10-0-135-121 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:03:56.706824 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.706810    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:03:56.707064 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.706935    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:03:56.711730 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.711699    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:03:56.713303 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.713290    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ljzk4"
Jul 27 21:03:56.713812 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.712807    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-121.ec2.internal.18c6416c88bbfa42  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-121.ec2.internal,UID:ip-10-0-135-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-121.ec2.internal,},FirstTimestamp:2026-07-27 21:03:56.705274434 +0000 UTC m=+0.375766072,LastTimestamp:2026-07-27 21:03:56.705274434 +0000 UTC m=+0.375766072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-121.ec2.internal,}"
Jul 27 21:03:56.717081 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.717064    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:03:56.717081 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.717074    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:03:56.717567 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.717311    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:56.718032 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718017    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:03:56.718117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718034    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:03:56.718117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718050    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:03:56.718208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718127    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:03:56.718208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718136    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:03:56.718208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718199    2555 factory.go:55] Registering systemd factory
Jul 27 21:03:56.718402 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718210    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:03:56.718451 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718413    2555 factory.go:153] Registering CRI-O factory
Jul 27 21:03:56.718451 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718421    2555 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:03:56.718535 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718462    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:03:56.718535 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718480    2555 factory.go:103] Registering Raw factory
Jul 27 21:03:56.718535 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.718491    2555 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:03:56.719405 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.719381    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:03:56.719507 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.719490    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:03:56.719665 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.719649    2555 manager.go:349] Starting recovery of all containers
Jul 27 21:03:56.730409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.730392    2555 manager.go:354] Recovery completed
Jul 27 21:03:56.735399 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.735385    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.737077 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737059    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.737140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737094    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.737140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737108    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.737618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737603    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:03:56.737618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737617    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:03:56.737738 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737633    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:03:56.737738 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.737632    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.738290 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.738274    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.738356 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.738310    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.738356 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.738326    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.739506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.739493    2555 policy_none.go:49] "None policy: Start"
Jul 27 21:03:56.739561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.739514    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:03:56.739561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.739525    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:03:56.740456 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.740446    2555 policy_none.go:47] "Start"
Jul 27 21:03:56.775391 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775277    2555 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:03:56.775472 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.775410    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:03:56.775472 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775423    2555 server.go:87] "Starting device plugin registration server"
Jul 27 21:03:56.775615 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775604    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:03:56.775672 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775617    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:03:56.775867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775842    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:03:56.775955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775933    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:03:56.775955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.775941    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:03:56.776332 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.776301    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:03:56.776422 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.776344    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:56.850810 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.850779    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:03:56.851937 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.851923    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:03:56.852000 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.851957    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:03:56.852000 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.851979    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:03:56.852098 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.852013    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:03:56.854472 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.854453    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:56.876328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.876275    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.877091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.877075    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.877172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.877103    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.877172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.877115    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.877172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.877137    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.882861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.882849    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.882913 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.882868    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-121.ec2.internal\": node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:56.897456 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.897441    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:56.952900 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.952876    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"]
Jul 27 21:03:56.953001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.952943    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.953765 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.953738    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.953839 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.953782    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.953839 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.953792    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.954878 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.954865    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.954997 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.954983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.955029 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955017    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.955542 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955525    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.955636 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955549    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.955636 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955565    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.955636 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955605    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.955636 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955627    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.955781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.955639    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.956730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.956717    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.956785 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.956741    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:56.957369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.957351    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:56.957447 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.957373    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:56.957447 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:56.957386    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:56.975306 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.975284    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-121.ec2.internal\" not found" node="ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.979303 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.979287    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-121.ec2.internal\" not found" node="ip-10-0-135-121.ec2.internal"
Jul 27 21:03:56.997640 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:56.997623    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.020090 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.020064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.020192 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.020095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.020192 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.020112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33a67532ab234867bd2a1f4dcc4b742b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-121.ec2.internal\" (UID: \"33a67532ab234867bd2a1f4dcc4b742b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.097721 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.097687    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.121151 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.121207 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.121241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.121241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33a67532ab234867bd2a1f4dcc4b742b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-121.ec2.internal\" (UID: \"33a67532ab234867bd2a1f4dcc4b742b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.121305 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33a67532ab234867bd2a1f4dcc4b742b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-121.ec2.internal\" (UID: \"33a67532ab234867bd2a1f4dcc4b742b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.121305 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.121278    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82507c9c83a71fa5aee586b264127447-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal\" (UID: \"82507c9c83a71fa5aee586b264127447\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.198544 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.198483    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.278038 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.278016    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.283404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.283388    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:57.298837 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.298809    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.399382 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.399346    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.499914 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.499851    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.600438 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.600403    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.649969 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.649938    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:03:57.650357 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.650105    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:03:57.701443 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.701407    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.717485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.717437    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:58:56 +0000 UTC" deadline="2028-04-13 03:24:57.492712284 +0000 UTC"
Jul 27 21:03:57.717485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.717478    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15006h20m59.775238366s"
Jul 27 21:03:57.801817 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.801785    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:57.891532 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:57.891499    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82507c9c83a71fa5aee586b264127447.slice/crio-ceaa8d250668caab861d3c171214f656c36fd3fd63bb1053089d130328fa58cb WatchSource:0}: Error finding container ceaa8d250668caab861d3c171214f656c36fd3fd63bb1053089d130328fa58cb: Status 404 returned error can't find the container with id ceaa8d250668caab861d3c171214f656c36fd3fd63bb1053089d130328fa58cb
Jul 27 21:03:57.891924 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:57.891905    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a67532ab234867bd2a1f4dcc4b742b.slice/crio-1e1029d741e38f6dd94790035813e99da020efebfcc0738a8f35c1573927f6cc WatchSource:0}: Error finding container 1e1029d741e38f6dd94790035813e99da020efebfcc0738a8f35c1573927f6cc: Status 404 returned error can't find the container with id 1e1029d741e38f6dd94790035813e99da020efebfcc0738a8f35c1573927f6cc
Jul 27 21:03:57.898393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:57.898378    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:03:57.902019 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:57.902002    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:58.002382 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.002350    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:58.102988 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.102901    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-121.ec2.internal\" not found"
Jul 27 21:03:58.119638 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.119620    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:58.166023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.166002    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:58.217536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.217512    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:58.227367 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.227344    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:03:58.229714 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.229701    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal"
Jul 27 21:03:58.241014 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.240995    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:03:58.268101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.268078    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:58.693547 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.693503    2555 apiserver.go:52] "Watching apiserver"
Jul 27 21:03:58.704938 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.704900    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:03:58.705298 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.705275    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2","openshift-image-registry/node-ca-z6nvg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal","openshift-multus/multus-rwl5b","openshift-multus/network-metrics-daemon-gb86p","openshift-network-operator/iptables-alerter-qk2hr","kube-system/konnectivity-agent-jxdts","openshift-cluster-node-tuning-operator/tuned-7xlnp","openshift-dns/node-resolver-5f2l4","openshift-multus/multus-additional-cni-plugins-2c5z6","openshift-network-diagnostics/network-check-target-d9xcx","openshift-ovn-kubernetes/ovnkube-node-926vh","kube-system/global-pull-secret-syncer-wffr5","kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal"]
Jul 27 21:03:58.707795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.707769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.710272 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.710251    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9jplt\""
Jul 27 21:03:58.710519 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.710503    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:03:58.710687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.710674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:03:58.711451 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.711428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:03:58.711573 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.711538    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:03:58.712242 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.712215    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.714490 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.714469    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.714649 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.714609    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.714925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.714906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.715001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.714975    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4dmt\""
Jul 27 21:03:58.718520 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.717223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.718520 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.717497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.718520 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.717563    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:03:58.718520 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.717611    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:03:58.719385 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.719366    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:03:58.719598 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.719583    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gspzp\""
Jul 27 21:03:58.719784 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.719767    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:03:58.719951 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.719937    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.720240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.720222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.722670 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.722648    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.723398 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.723377    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:03:58.723643 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.723609    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:03:58.723718 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.723681    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-djnzn\""
Jul 27 21:03:58.723867 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.723852    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.724334 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.724316    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:03:58.724525 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.724510    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.724690 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.724678    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:03:58.726373 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.725347    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.726373 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.726149    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t9p8x\""
Jul 27 21:03:58.726373 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.726173    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.726566 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.726426    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.728096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.727228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.728096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.727869    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.728096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.728091    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:03:58.728460 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.728361    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.728460 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.728414    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-trrwd\""
Jul 27 21:03:58.729650 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.729632    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.729852 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.729634    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.729934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.729896    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xc7tp\""
Jul 27 21:03:58.730159 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.730141    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.730495 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.730393    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-bin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731473    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-conf-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-etc-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-netd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731589    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-ovn\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731611    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2bh\" (UniqueName: \"kubernetes.io/projected/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-kube-api-access-dn2bh\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731638    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-run\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-sys\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731696    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-var-lib-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731957    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.731983    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-script-lib\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-host\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-netns\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.732799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-etc-kubernetes\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-systemd-units\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-kubernetes\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-kubelet\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqd4t\" (UniqueName: \"kubernetes.io/projected/585a203a-d420-4b43-b6f4-248884bede2b-kube-api-access-sqd4t\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732267    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-kubelet\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-node-log\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732315    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-log-socket\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-tmp\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mzw\" (UniqueName: \"kubernetes.io/projected/da8288fe-6985-48bb-ad34-0cd66e4499b3-kube-api-access-h2mzw\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-os-release\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-netns\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-var-lib-kubelet\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732512    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732530    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-multus-certs\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732550    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.733635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-system-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cni-binary-copy\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-lib-modules\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-daemon-config\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-multus\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732715    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-tuned\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovn-node-metrics-cert\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732770    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6k78d\""
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/256d5c0d-52d6-4201-abc3-0354dada8960-agent-certs\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/256d5c0d-52d6-4201-abc3-0354dada8960-konnectivity-ca\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-config\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732898    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-slash\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysconfig\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-socket-dir-parent\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.732974    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-k8s-cni-cncf-io\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-hostroot\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733028    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:03:58.734445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733046    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-conf\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-systemd\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z9h\" (UniqueName: \"kubernetes.io/projected/cd5121fa-06dc-40f8-844e-7b1237fbffa9-kube-api-access-w2z9h\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733155    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-systemd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-modprobe-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.733224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.734725    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.734816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cnibin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.735393 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.735384    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-bin\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.735842 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.735622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-env-overrides\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.736021 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.736001    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.736098 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.736077    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:03:58.737548 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.737344    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:03:58.738036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.738016    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:03:58.738275 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.738256    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:03:58.738366 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.738348    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvtfq\""
Jul 27 21:03:58.749419 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.749360    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w75qk"
Jul 27 21:03:58.759936 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.759914    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w75qk"
Jul 27 21:03:58.819695 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.819664    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:03:58.836219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836179    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.836219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysconfig\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-socket-dir-parent\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-k8s-cni-cncf-io\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-hostroot\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836344    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-conf\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-systemd\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z9h\" (UniqueName: \"kubernetes.io/projected/cd5121fa-06dc-40f8-844e-7b1237fbffa9-kube-api-access-w2z9h\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-systemd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836434    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-modprobe-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cnibin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-bin\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836526    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-env-overrides\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836551    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-bin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836626    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-conf-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-etc-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-netd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836718    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-ovn\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.836797 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2bh\" (UniqueName: \"kubernetes.io/projected/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-kube-api-access-dn2bh\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-run\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-sys\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-var-lib-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-script-lib\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.836988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-registration-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5t7\" (UniqueName: \"kubernetes.io/projected/c18b8d6e-6d55-4925-affb-96030d7814e3-kube-api-access-gz5t7\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837040    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-host\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysconfig\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-systemd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-socket-dir-parent\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837268    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-bin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837293    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-k8s-cni-cncf-io\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-hostroot\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-netns\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-etc-kubernetes\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-conf\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-systemd-units\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-host\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-modprobe-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8699b98e-a1d2-4779-9534-29fddb721504-hosts-file\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-netns\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-run\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-etc-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-etc-kubernetes\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837513    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dmh\" (UniqueName: \"kubernetes.io/projected/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-kube-api-access-d2dmh\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837556    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-systemd-units\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-conf-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.837828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-run-ovn\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-bin\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-cni-netd\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837632    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-systemd\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-var-lib-openvswitch\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-kubelet-config\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-sys\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837732    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-kubernetes\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cnibin\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-kubernetes\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-kubelet\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqd4t\" (UniqueName: \"kubernetes.io/projected/585a203a-d420-4b43-b6f4-248884bede2b-kube-api-access-sqd4t\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-kubelet\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cff53865-e99c-4785-886a-13efb209469f-iptables-alerter-script\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837942    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcgd\" (UniqueName: \"kubernetes.io/projected/8699b98e-a1d2-4779-9534-29fddb721504-kube-api-access-xtcgd\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-kubelet\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.838660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.837968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-system-cni-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-cnibin\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-kubelet\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-node-log\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-log-socket\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-tmp\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mzw\" (UniqueName: \"kubernetes.io/projected/da8288fe-6985-48bb-ad34-0cd66e4499b3-kube-api-access-h2mzw\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-os-release\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-script-lib\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-netns\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838261    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-os-release\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-var-lib-kubelet\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-run-netns\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-multus-certs\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-log-socket\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-run-multus-certs\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-node-log\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.839389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838399    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-sys-fs\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8699b98e-a1d2-4779-9534-29fddb721504-tmp-dir\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838447    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-host\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838465    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-var-lib-kubelet\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-os-release\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-dbus\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-system-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cni-binary-copy\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-sysctl-d\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838665    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-lib-modules\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-env-overrides\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-daemon-config\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-multus\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-system-cni-dir\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-tuned\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.840240 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8288fe-6985-48bb-ad34-0cd66e4499b3-lib-modules\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838872    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd5121fa-06dc-40f8-844e-7b1237fbffa9-host-var-lib-cni-multus\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovn-node-metrics-cert\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.838952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/256d5c0d-52d6-4201-abc3-0354dada8960-agent-certs\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/256d5c0d-52d6-4201-abc3-0354dada8960-konnectivity-ca\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-device-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-config\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-socket-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-cni-binary-copy\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcw8m\" (UniqueName: \"kubernetes.io/projected/d718875b-1fc1-45cd-8971-7b64e32c00f5-kube-api-access-wcw8m\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839320    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd5121fa-06dc-40f8-844e-7b1237fbffa9-multus-daemon-config\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839359    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cff53865-e99c-4785-886a-13efb209469f-host-slash\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-slash\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7nj\" (UniqueName: \"kubernetes.io/projected/cff53865-e99c-4785-886a-13efb209469f-kube-api-access-gz7nj\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.841093 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-host-slash\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.841864 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-serviceca\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.841864 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/256d5c0d-52d6-4201-abc3-0354dada8960-konnectivity-ca\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.841864 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.839689    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:58.841864 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.839770    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:03:59.339726697 +0000 UTC m=+3.010218337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:58.841864 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.839839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovnkube-config\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.842909 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.842609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-tmp\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.843199 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.843150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/256d5c0d-52d6-4201-abc3-0354dada8960-agent-certs\") pod \"konnectivity-agent-jxdts\" (UID: \"256d5c0d-52d6-4201-abc3-0354dada8960\") " pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:58.843290 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.843244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-ovn-node-metrics-cert\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.843880 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.843850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da8288fe-6985-48bb-ad34-0cd66e4499b3-etc-tuned\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.846838 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.846745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z9h\" (UniqueName: \"kubernetes.io/projected/cd5121fa-06dc-40f8-844e-7b1237fbffa9-kube-api-access-w2z9h\") pod \"multus-rwl5b\" (UID: \"cd5121fa-06dc-40f8-844e-7b1237fbffa9\") " pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:58.852193 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.852164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mzw\" (UniqueName: \"kubernetes.io/projected/da8288fe-6985-48bb-ad34-0cd66e4499b3-kube-api-access-h2mzw\") pod \"tuned-7xlnp\" (UID: \"da8288fe-6985-48bb-ad34-0cd66e4499b3\") " pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:58.853102 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.853081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqd4t\" (UniqueName: \"kubernetes.io/projected/585a203a-d420-4b43-b6f4-248884bede2b-kube-api-access-sqd4t\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:58.853412 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.853370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2bh\" (UniqueName: \"kubernetes.io/projected/6cb44840-d1e7-4c34-8192-e2e93f2c1e55-kube-api-access-dn2bh\") pod \"ovnkube-node-926vh\" (UID: \"6cb44840-d1e7-4c34-8192-e2e93f2c1e55\") " pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:58.854208 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.854188    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:58.854298 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.854213    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:58.854298 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.854227    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:58.854298 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.854287    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:59.354269847 +0000 UTC m=+3.024761486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:58.857543 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.857498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal" event={"ID":"82507c9c83a71fa5aee586b264127447","Type":"ContainerStarted","Data":"ceaa8d250668caab861d3c171214f656c36fd3fd63bb1053089d130328fa58cb"}
Jul 27 21:03:58.858826 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.858803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal" event={"ID":"33a67532ab234867bd2a1f4dcc4b742b","Type":"ContainerStarted","Data":"1e1029d741e38f6dd94790035813e99da020efebfcc0738a8f35c1573927f6cc"}
Jul 27 21:03:58.940223 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcgd\" (UniqueName: \"kubernetes.io/projected/8699b98e-a1d2-4779-9534-29fddb721504-kube-api-access-xtcgd\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.940382 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-system-cni-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940382 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-cnibin\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940382 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-cnibin\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940382 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-system-cni-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940433    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-sys-fs\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8699b98e-a1d2-4779-9534-29fddb721504-tmp-dir\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940480    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-host\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-os-release\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-sys-fs\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940526    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-dbus\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-host\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.940593 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940578    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-os-release\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-dbus\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-device-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-socket-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-device-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940799    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcw8m\" (UniqueName: \"kubernetes.io/projected/d718875b-1fc1-45cd-8971-7b64e32c00f5-kube-api-access-wcw8m\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940854    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8699b98e-a1d2-4779-9534-29fddb721504-tmp-dir\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cff53865-e99c-4785-886a-13efb209469f-host-slash\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cff53865-e99c-4785-886a-13efb209469f-host-slash\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7nj\" (UniqueName: \"kubernetes.io/projected/cff53865-e99c-4785-886a-13efb209469f-kube-api-access-gz7nj\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-serviceca\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941031 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.940994    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-socket-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c18b8d6e-6d55-4925-affb-96030d7814e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-registration-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5t7\" (UniqueName: \"kubernetes.io/projected/c18b8d6e-6d55-4925-affb-96030d7814e3-kube-api-access-gz5t7\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8699b98e-a1d2-4779-9534-29fddb721504-hosts-file\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.941413    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941457    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c18b8d6e-6d55-4925-affb-96030d7814e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:58.941466    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:59.441448888 +0000 UTC m=+3.111940513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dmh\" (UniqueName: \"kubernetes.io/projected/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-kube-api-access-d2dmh\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941487    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-serviceca\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-kubelet-config\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.941861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cff53865-e99c-4785-886a-13efb209469f-iptables-alerter-script\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.942488 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941553    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/639af4e4-909c-4f58-b7ca-f1c2ba762f03-kubelet-config\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:58.942488 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941411    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d718875b-1fc1-45cd-8971-7b64e32c00f5-registration-dir\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.942488 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.941578    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8699b98e-a1d2-4779-9534-29fddb721504-hosts-file\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.942488 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.942071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cff53865-e99c-4785-886a-13efb209469f-iptables-alerter-script\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.948317 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.948264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcgd\" (UniqueName: \"kubernetes.io/projected/8699b98e-a1d2-4779-9534-29fddb721504-kube-api-access-xtcgd\") pod \"node-resolver-5f2l4\" (UID: \"8699b98e-a1d2-4779-9534-29fddb721504\") " pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:58.951994 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.951976    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcw8m\" (UniqueName: \"kubernetes.io/projected/d718875b-1fc1-45cd-8971-7b64e32c00f5-kube-api-access-wcw8m\") pod \"aws-ebs-csi-driver-node-xdql2\" (UID: \"d718875b-1fc1-45cd-8971-7b64e32c00f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:58.954920 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.954899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7nj\" (UniqueName: \"kubernetes.io/projected/cff53865-e99c-4785-886a-13efb209469f-kube-api-access-gz7nj\") pod \"iptables-alerter-qk2hr\" (UID: \"cff53865-e99c-4785-886a-13efb209469f\") " pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:58.955537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.955384    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5t7\" (UniqueName: \"kubernetes.io/projected/c18b8d6e-6d55-4925-affb-96030d7814e3-kube-api-access-gz5t7\") pod \"multus-additional-cni-plugins-2c5z6\" (UID: \"c18b8d6e-6d55-4925-affb-96030d7814e3\") " pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:58.955537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:58.955479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dmh\" (UniqueName: \"kubernetes.io/projected/b32691ec-ae72-4d5b-a088-ed2b704d6b4d-kube-api-access-d2dmh\") pod \"node-ca-z6nvg\" (UID: \"b32691ec-ae72-4d5b-a088-ed2b704d6b4d\") " pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:59.021419 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.021387    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:03:59.031649 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.031623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xlnp"
Jul 27 21:03:59.043428 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.043407    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rwl5b"
Jul 27 21:03:59.048171 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.048152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:03:59.061656 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.061628    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f2l4"
Jul 27 21:03:59.069461 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.069433    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2"
Jul 27 21:03:59.077202 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.077183    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6nvg"
Jul 27 21:03:59.084905 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.084884    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2c5z6"
Jul 27 21:03:59.090419 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.090400    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qk2hr"
Jul 27 21:03:59.194199 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.194169    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:59.346017 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.345935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:03:59.346174 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.346098    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:59.346174 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.346166    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:04:00.346148444 +0000 UTC m=+4.016640076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:59.446448 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.446411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.446486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446584    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446603    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446625    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446633    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:00.446620843 +0000 UTC m=+4.117112468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:59.446637 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446639    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:59.446905 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:03:59.446697    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:00.446681604 +0000 UTC m=+4.117173243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:59.764310 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.764208    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:58 +0000 UTC" deadline="2028-01-10 03:43:40.110863815 +0000 UTC"
Jul 27 21:03:59.764310 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:03:59.764277    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12750h39m40.346600207s"
Jul 27 21:03:59.905144 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.904995    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff53865_e99c_4785_886a_13efb209469f.slice/crio-4a2e7d8dd3d980f930238d25f8af0431faaa05e0fa3d9ab4138566d7b3f0095b WatchSource:0}: Error finding container 4a2e7d8dd3d980f930238d25f8af0431faaa05e0fa3d9ab4138566d7b3f0095b: Status 404 returned error can't find the container with id 4a2e7d8dd3d980f930238d25f8af0431faaa05e0fa3d9ab4138566d7b3f0095b
Jul 27 21:03:59.911452 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.911433    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8699b98e_a1d2_4779_9534_29fddb721504.slice/crio-0d4b1c4b1d3f9deb4fe979685baf3eeebc8b0b6f8ff657e19a276da875b1a253 WatchSource:0}: Error finding container 0d4b1c4b1d3f9deb4fe979685baf3eeebc8b0b6f8ff657e19a276da875b1a253: Status 404 returned error can't find the container with id 0d4b1c4b1d3f9deb4fe979685baf3eeebc8b0b6f8ff657e19a276da875b1a253
Jul 27 21:03:59.912969 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.912937    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256d5c0d_52d6_4201_abc3_0354dada8960.slice/crio-0543d625e8e5943fad66245ec22b8c62f34b56e4e15f191353634a2b9dd5bb24 WatchSource:0}: Error finding container 0543d625e8e5943fad66245ec22b8c62f34b56e4e15f191353634a2b9dd5bb24: Status 404 returned error can't find the container with id 0543d625e8e5943fad66245ec22b8c62f34b56e4e15f191353634a2b9dd5bb24
Jul 27 21:03:59.914054 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.913728    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18b8d6e_6d55_4925_affb_96030d7814e3.slice/crio-27edc8e1428535a8921ec4c8da612fb9ecebf5cfa2b0f79a03ccef5932e2b9df WatchSource:0}: Error finding container 27edc8e1428535a8921ec4c8da612fb9ecebf5cfa2b0f79a03ccef5932e2b9df: Status 404 returned error can't find the container with id 27edc8e1428535a8921ec4c8da612fb9ecebf5cfa2b0f79a03ccef5932e2b9df
Jul 27 21:03:59.914214 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.914191    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5121fa_06dc_40f8_844e_7b1237fbffa9.slice/crio-4bcaca0dc108b746a1ba0791f4f35523a5960555b8b08ebb221368797e11db75 WatchSource:0}: Error finding container 4bcaca0dc108b746a1ba0791f4f35523a5960555b8b08ebb221368797e11db75: Status 404 returned error can't find the container with id 4bcaca0dc108b746a1ba0791f4f35523a5960555b8b08ebb221368797e11db75
Jul 27 21:03:59.915634 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.915546    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32691ec_ae72_4d5b_a088_ed2b704d6b4d.slice/crio-9fab5a1c0c5690fb26ba1b947230404c0b396f6b670f12ed24c611f876f2489d WatchSource:0}: Error finding container 9fab5a1c0c5690fb26ba1b947230404c0b396f6b670f12ed24c611f876f2489d: Status 404 returned error can't find the container with id 9fab5a1c0c5690fb26ba1b947230404c0b396f6b670f12ed24c611f876f2489d
Jul 27 21:03:59.916039 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.915960    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8288fe_6985_48bb_ad34_0cd66e4499b3.slice/crio-5629826af5159fe5113a688f8f96b2d526112172488ee3404989c9def96e92df WatchSource:0}: Error finding container 5629826af5159fe5113a688f8f96b2d526112172488ee3404989c9def96e92df: Status 404 returned error can't find the container with id 5629826af5159fe5113a688f8f96b2d526112172488ee3404989c9def96e92df
Jul 27 21:03:59.916614 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.916538    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb44840_d1e7_4c34_8192_e2e93f2c1e55.slice/crio-6d3e7ba0f9433f954c0540120cc5265d6ef01cd63d6f96ecad903d3cdf4d642f WatchSource:0}: Error finding container 6d3e7ba0f9433f954c0540120cc5265d6ef01cd63d6f96ecad903d3cdf4d642f: Status 404 returned error can't find the container with id 6d3e7ba0f9433f954c0540120cc5265d6ef01cd63d6f96ecad903d3cdf4d642f
Jul 27 21:03:59.917975 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:03:59.917956    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd718875b_1fc1_45cd_8971_7b64e32c00f5.slice/crio-9cb3fc097f68bd042edc1c3b7c84928135c372c3c923cc4c1c63f435336272e1 WatchSource:0}: Error finding container 9cb3fc097f68bd042edc1c3b7c84928135c372c3c923cc4c1c63f435336272e1: Status 404 returned error can't find the container with id 9cb3fc097f68bd042edc1c3b7c84928135c372c3c923cc4c1c63f435336272e1
Jul 27 21:04:00.351940 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.351845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:00.352076 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.352009    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:00.352076 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.352069    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:04:02.352052543 +0000 UTC m=+6.022544172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:00.452263 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.452127    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:00.452263 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.452218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:00.452521 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452348    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:00.452521 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452403    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:02.452384947 +0000 UTC m=+6.122876588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:00.452835 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452795    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:00.452835 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452820    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:00.452835 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452833    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:00.453016 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.452866    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:02.452854621 +0000 UTC m=+6.123346252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:00.765447 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.765339    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:58 +0000 UTC" deadline="2028-04-06 12:31:26.06277785 +0000 UTC"
Jul 27 21:04:00.765447 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.765407    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14847h27m25.297375428s"
Jul 27 21:04:00.855483 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.855456    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:00.855629 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.855582    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:00.856331 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.856311    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:00.856437 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.856418    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:00.856502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.856481    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:00.856564 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:00.856547    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:00.884108 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.884079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rwl5b" event={"ID":"cd5121fa-06dc-40f8-844e-7b1237fbffa9","Type":"ContainerStarted","Data":"4bcaca0dc108b746a1ba0791f4f35523a5960555b8b08ebb221368797e11db75"}
Jul 27 21:04:00.887205 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.887129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerStarted","Data":"27edc8e1428535a8921ec4c8da612fb9ecebf5cfa2b0f79a03ccef5932e2b9df"}
Jul 27 21:04:00.894472 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.894395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qk2hr" event={"ID":"cff53865-e99c-4785-886a-13efb209469f","Type":"ContainerStarted","Data":"4a2e7d8dd3d980f930238d25f8af0431faaa05e0fa3d9ab4138566d7b3f0095b"}
Jul 27 21:04:00.898742 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.898129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal" event={"ID":"33a67532ab234867bd2a1f4dcc4b742b","Type":"ContainerStarted","Data":"33828cb4e51dd3b71b39c88be2030aac988a8a3acbc2ee4d533e38b1a4f04328"}
Jul 27 21:04:00.912298 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.912213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2" event={"ID":"d718875b-1fc1-45cd-8971-7b64e32c00f5","Type":"ContainerStarted","Data":"9cb3fc097f68bd042edc1c3b7c84928135c372c3c923cc4c1c63f435336272e1"}
Jul 27 21:04:00.917235 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.917141    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-121.ec2.internal" podStartSLOduration=2.917125468 podStartE2EDuration="2.917125468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:00.916095439 +0000 UTC m=+4.586587088" watchObservedRunningTime="2026-07-27 21:04:00.917125468 +0000 UTC m=+4.587617118"
Jul 27 21:04:00.921735 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.921650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"6d3e7ba0f9433f954c0540120cc5265d6ef01cd63d6f96ecad903d3cdf4d642f"}
Jul 27 21:04:00.923733 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.923677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xlnp" event={"ID":"da8288fe-6985-48bb-ad34-0cd66e4499b3","Type":"ContainerStarted","Data":"5629826af5159fe5113a688f8f96b2d526112172488ee3404989c9def96e92df"}
Jul 27 21:04:00.926046 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.925996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jxdts" event={"ID":"256d5c0d-52d6-4201-abc3-0354dada8960","Type":"ContainerStarted","Data":"0543d625e8e5943fad66245ec22b8c62f34b56e4e15f191353634a2b9dd5bb24"}
Jul 27 21:04:00.928619 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.928518    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f2l4" event={"ID":"8699b98e-a1d2-4779-9534-29fddb721504","Type":"ContainerStarted","Data":"0d4b1c4b1d3f9deb4fe979685baf3eeebc8b0b6f8ff657e19a276da875b1a253"}
Jul 27 21:04:00.934771 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.934236    2555 generic.go:356] "Generic (PLEG): container finished" podID="82507c9c83a71fa5aee586b264127447" containerID="f8722a7c43ad93df09d039d2e40ad9963a387e1e7f6fdd17a47ee67b0811fe76" exitCode=0
Jul 27 21:04:00.934771 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.934304    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal" event={"ID":"82507c9c83a71fa5aee586b264127447","Type":"ContainerDied","Data":"f8722a7c43ad93df09d039d2e40ad9963a387e1e7f6fdd17a47ee67b0811fe76"}
Jul 27 21:04:00.937416 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:00.937391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6nvg" event={"ID":"b32691ec-ae72-4d5b-a088-ed2b704d6b4d","Type":"ContainerStarted","Data":"9fab5a1c0c5690fb26ba1b947230404c0b396f6b670f12ed24c611f876f2489d"}
Jul 27 21:04:01.947121 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:01.947053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal" event={"ID":"82507c9c83a71fa5aee586b264127447","Type":"ContainerStarted","Data":"1d5c10c889b9a42ae5c3ffacc64af0b892b468f84b68f012cd88d7f3a2f41b03"}
Jul 27 21:04:01.963726 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:01.962586    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-121.ec2.internal" podStartSLOduration=3.96256825 podStartE2EDuration="3.96256825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:01.962208539 +0000 UTC m=+5.632700186" watchObservedRunningTime="2026-07-27 21:04:01.96256825 +0000 UTC m=+5.633059899"
Jul 27 21:04:02.366628 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.366596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:02.366871 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.366853    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:02.366955 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.366919    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:04:06.366899611 +0000 UTC m=+10.037391244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:02.467847 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.467808    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:02.468001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.467879    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:02.468074 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468019    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:02.468074 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468042    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:02.468074 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468055    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:02.468227 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468112    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:06.468094952 +0000 UTC m=+10.138586595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:02.468383 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468313    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:02.468383 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.468366    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:06.468353508 +0000 UTC m=+10.138845137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:02.852793 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.852764    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:02.852960 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.852801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:02.852960 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.852906    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:02.853082 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.853031    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:02.853082 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:02.852672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:02.853230 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:02.853175    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:04.852616 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:04.852579    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:04.853037 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:04.852715    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:04.853037 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:04.852728    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:04.853037 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:04.852831    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:04.853037 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:04.852883    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:04.853037 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:04.852991    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:06.402647 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.402604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:06.403155 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.402784    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:06.403155 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.402862    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.402838837 +0000 UTC m=+18.073330479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:06.504086 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.504042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:06.504262 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.504129    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:06.504329 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504271    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:06.504392 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504351    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.504312939 +0000 UTC m=+18.174804568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:06.504807 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504782    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:06.504807 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504804    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:06.504969 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504817    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:06.504969 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.504858    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.504843987 +0000 UTC m=+18.175335621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:06.853774 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.853212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:06.855099 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.854218    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:06.855099 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.854297    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:06.855099 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.854381    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:06.855099 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:06.854424    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:06.855099 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:06.854492    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:08.852898 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:08.852861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:08.853371 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:08.852996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:08.853464 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:08.853445    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:08.853565 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:08.853542    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:08.853898 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:08.853708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:08.853898 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:08.853856    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:10.852201 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:10.852173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:10.852201 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:10.852206    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:10.852682 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:10.852173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:10.852682 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:10.852302    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:10.852682 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:10.852436    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:10.852682 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:10.852521    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:12.852675 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:12.852639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:12.852675 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:12.852667    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:12.853202 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:12.852639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:12.853202 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:12.852787    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:12.853202 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:12.852859    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:12.853202 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:12.852948    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:14.464919 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.464877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:14.465373 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.465044    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:14.465373 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.465132    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:04:30.465101299 +0000 UTC m=+34.135592927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:14.566026 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.565976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:14.566215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.566069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:14.566215 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566153    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:14.566215 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566189    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:14.566215 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566203    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:14.566413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566157    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:14.566413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566264    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:30.56624339 +0000 UTC m=+34.236735034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:14.566413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.566300    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:30.566285313 +0000 UTC m=+34.236776942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:14.852654 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.852615    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:14.852855 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.852618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:14.852855 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:14.852776    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:14.852977 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.852739    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:14.852977 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.852875    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:14.852977 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:14.852932    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:16.853828 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:16.853799    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:16.854281 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:16.853889    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:16.854281 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:16.853928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:16.854281 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:16.853948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:16.854281 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:16.854053    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:16.854281 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:16.854140    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:18.852884 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.852572    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:18.853540 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.852573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:18.853540 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:18.852970    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:18.853540 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.852574    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:18.853540 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:18.853053    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:18.853540 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:18.853153    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:18.977807 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.977778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2" event={"ID":"d718875b-1fc1-45cd-8971-7b64e32c00f5","Type":"ContainerStarted","Data":"0da8f78d7bd2c976a24188726f459caaf8d8c36b383d664e51b4de2cd5b1c9d9"}
Jul 27 21:04:18.980452 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980430    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"160049f7e835dda011818e1b65c3f94d6ec891dee2acc6f6b468738dcb3a0152"}
Jul 27 21:04:18.980565 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"a4d7e87d64e9afe79ec389640cb93ed34376288b8a794cbc3c0d19c772076085"}
Jul 27 21:04:18.980565 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"4354ef35ef118f273d355e118f8556d0b73866611a0631c2781d544451b60e98"}
Jul 27 21:04:18.980565 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980557    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"17597761a74381ae5a13e9c9f072b436ce15de332aed7e968e43f02f5b6037d9"}
Jul 27 21:04:18.980684 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"c4bc188f6a67246335d659f2716f13fb68245c98f4a34b0387a86a476fb41e81"}
Jul 27 21:04:18.980684 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.980617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"d1db872e42fe0cc656f6815405819a461b1abae311d26630b21f94b5bb59badd"}
Jul 27 21:04:18.981932 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.981909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xlnp" event={"ID":"da8288fe-6985-48bb-ad34-0cd66e4499b3","Type":"ContainerStarted","Data":"cd5311b0b136231415df3a42657c72c0fad43e06e32f7b176080e5c56bfa574e"}
Jul 27 21:04:18.983352 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.983332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jxdts" event={"ID":"256d5c0d-52d6-4201-abc3-0354dada8960","Type":"ContainerStarted","Data":"df48d16fe0c4e1906734667a33c178afbeff5fc3df66d4e8d6aee59e360f79b2"}
Jul 27 21:04:18.984694 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.984673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f2l4" event={"ID":"8699b98e-a1d2-4779-9534-29fddb721504","Type":"ContainerStarted","Data":"64b8da671b6c703d6a1de09dc096d31d0e9b5862446c6a7bae27ee5efbbc97b3"}
Jul 27 21:04:18.986074 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.986053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6nvg" event={"ID":"b32691ec-ae72-4d5b-a088-ed2b704d6b4d","Type":"ContainerStarted","Data":"3f224a38d7e8348ec34cfeedcb8daa6764c5c000414879c4ada12d1c353627c9"}
Jul 27 21:04:18.987372 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.987353    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rwl5b" event={"ID":"cd5121fa-06dc-40f8-844e-7b1237fbffa9","Type":"ContainerStarted","Data":"b6701ec98bf1dda706fe484790936e00e6ed9a73c8dc42ea3834cb1754643a1e"}
Jul 27 21:04:18.988814 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.988790    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="ec27c8d774d93cebd6e4e61c45a201016936acbda095a45e6d2db27f3a07d618" exitCode=0
Jul 27 21:04:18.988898 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.988823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"ec27c8d774d93cebd6e4e61c45a201016936acbda095a45e6d2db27f3a07d618"}
Jul 27 21:04:18.999041 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:18.999009    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xlnp" podStartSLOduration=4.970639633 podStartE2EDuration="22.998998809s" totalImagesPullingTime="18.028359176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:18.998690023 +0000 UTC m=+22.669181670" watchObservedRunningTime="2026-07-27 21:04:18.998998809 +0000 UTC m=+22.669490462"
Jul 27 21:04:19.013207 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.013175    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6nvg" podStartSLOduration=4.961086115 podStartE2EDuration="23.013166896s" totalImagesPullingTime="18.052080781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.01288958 +0000 UTC m=+22.683381226" watchObservedRunningTime="2026-07-27 21:04:19.013166896 +0000 UTC m=+22.683658543"
Jul 27 21:04:19.070007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.069970    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rwl5b" podStartSLOduration=4.967359539 podStartE2EDuration="23.069959774s" totalImagesPullingTime="18.102600235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.069900834 +0000 UTC m=+22.740392480" watchObservedRunningTime="2026-07-27 21:04:19.069959774 +0000 UTC m=+22.740451420"
Jul 27 21:04:19.070121 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.070103    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jxdts" podStartSLOduration=9.060257262 podStartE2EDuration="23.07009852s" totalImagesPullingTime="14.009841258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.052581284 +0000 UTC m=+22.723072931" watchObservedRunningTime="2026-07-27 21:04:19.07009852 +0000 UTC m=+22.740590166"
Jul 27 21:04:19.085519 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.085485    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5f2l4" podStartSLOduration=5.02931998 podStartE2EDuration="23.08547461s" totalImagesPullingTime="18.05615463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.085274878 +0000 UTC m=+22.755766525" watchObservedRunningTime="2026-07-27 21:04:19.08547461 +0000 UTC m=+22.755966257"
Jul 27 21:04:19.363310 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.363285    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:04:19.790581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.790442    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:04:19.363305553Z","UUID":"d11b20bb-aff2-4041-a379-bf2dbbed7db5","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:04:19.792343 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.792320    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:04:19.792463 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.792361    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:04:19.992676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.992643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qk2hr" event={"ID":"cff53865-e99c-4785-886a-13efb209469f","Type":"ContainerStarted","Data":"c0cce4b5cae8238453d0b2204f3fdceca82ff28eb1efee9c3562a1fb6cf4475f"}
Jul 27 21:04:19.995289 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:19.995212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2" event={"ID":"d718875b-1fc1-45cd-8971-7b64e32c00f5","Type":"ContainerStarted","Data":"1d639e4d5afe2cfec9d08ac576affcfd6e086373b03ebc1f1ce8223c37d485ce"}
Jul 27 21:04:20.010403 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.010348    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qk2hr" podStartSLOduration=4.9484709 podStartE2EDuration="23.010331501s" totalImagesPullingTime="18.061860601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.010105017 +0000 UTC m=+23.680596665" watchObservedRunningTime="2026-07-27 21:04:20.010331501 +0000 UTC m=+23.680823150"
Jul 27 21:04:20.852869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.852837    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:20.853100 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.852837    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:20.853100 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.852965    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:20.853100 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:20.852968    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:20.853100 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:20.853043    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:20.853280 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:20.853127    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:20.875098 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.875041    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:04:20.875554 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.875537    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:04:20.999731 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:20.999694    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2" event={"ID":"d718875b-1fc1-45cd-8971-7b64e32c00f5","Type":"ContainerStarted","Data":"c39eb728805ffa271e1566d32cf3aa5f1f26b95322380223bde8da30917df5cd"}
Jul 27 21:04:21.003452 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:21.003427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"0ab08c3cd746751bd60cba01037bf05e32c448ae9db1228a422b07cb184ba80a"}
Jul 27 21:04:21.003785 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:21.003716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:04:21.004172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:21.004154    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jxdts"
Jul 27 21:04:21.020464 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:21.020422    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xdql2" podStartSLOduration=4.946671182 podStartE2EDuration="25.020407246s" totalImagesPullingTime="20.073736064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:21.020112304 +0000 UTC m=+24.690603951" watchObservedRunningTime="2026-07-27 21:04:21.020407246 +0000 UTC m=+24.690898893"
Jul 27 21:04:22.852535 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:22.852441    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:22.852928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:22.852497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:22.852928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:22.852630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:22.852928 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:22.852627    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:22.852928 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:22.852696    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:22.852928 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:22.852770    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:23.011507 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:23.011436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" event={"ID":"6cb44840-d1e7-4c34-8192-e2e93f2c1e55","Type":"ContainerStarted","Data":"ae5b3d9fb225d4cfd398f780ba2641bf26f931244dd92cb4dafd2d8ddcc6fc3e"}
Jul 27 21:04:23.043320 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:23.043279    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-926vh" podStartSLOduration=8.947192383 podStartE2EDuration="27.043263742s" totalImagesPullingTime="18.096071359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:23.04267827 +0000 UTC m=+26.713169930" watchObservedRunningTime="2026-07-27 21:04:23.043263742 +0000 UTC m=+26.713755389"
Jul 27 21:04:24.015436 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.015401    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="bd15c59deedd03ef361f50a5002760d7906a9fac89c3b0785085f126dbf5be58" exitCode=0
Jul 27 21:04:24.016204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.015490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"bd15c59deedd03ef361f50a5002760d7906a9fac89c3b0785085f126dbf5be58"}
Jul 27 21:04:24.016204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.015913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:24.016204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.015982    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:24.016204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.016030    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:24.030715 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.030585    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:24.031057 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.031041    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:24.162110 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.162085    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f2l4_8699b98e-a1d2-4779-9534-29fddb721504/dns-node-resolver/0.log"
Jul 27 21:04:24.853001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.852947    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:24.853001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.852961    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:24.853001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.852972    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:24.853209 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:24.853035    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:24.853409 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:24.853369    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:24.856218 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:24.853647    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:24.962868 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.962848    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gb86p"]
Jul 27 21:04:24.963319 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.963302    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d9xcx"]
Jul 27 21:04:24.966203 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:24.966186    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wffr5"]
Jul 27 21:04:25.017270 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:25.017249    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:25.017585 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:25.017257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:25.017585 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:25.017258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:25.017585 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:25.017357    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:25.017585 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:25.017438    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:25.017862 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:25.017841    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:25.318573 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:25.318551    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6nvg_b32691ec-ae72-4d5b-a088-ed2b704d6b4d/node-ca/0.log"
Jul 27 21:04:26.021928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:26.021896    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="a05e4b8ebde128f0f900cbd74bdf426f74b2988314060635c68fd4f7acaa94c7" exitCode=0
Jul 27 21:04:26.022272 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:26.021986    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"a05e4b8ebde128f0f900cbd74bdf426f74b2988314060635c68fd4f7acaa94c7"}
Jul 27 21:04:26.855905 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:26.855882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:26.856012 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:26.855979    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:26.856012 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:26.855997    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:26.856086 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:26.856072    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:26.856119 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:26.856108    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:26.856192 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:26.856176    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:28.031471 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:28.031286    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="9adcfd0cd862c412bcfb1a0ab498a5539e8078ea9ccbab4db341483ee2126d89" exitCode=0
Jul 27 21:04:28.031878 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:28.031365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"9adcfd0cd862c412bcfb1a0ab498a5539e8078ea9ccbab4db341483ee2126d89"}
Jul 27 21:04:28.853101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:28.853058    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:28.853101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:28.853092    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:28.853323 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:28.853190    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:28.853323 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:28.853276    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:28.853323 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:28.853319    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:28.853480 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:28.853462    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:30.497836 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.497799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:30.498273 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.497966    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:30.498273 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.498041    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs podName:585a203a-d420-4b43-b6f4-248884bede2b nodeName:}" failed. No retries permitted until 2026-07-27 21:05:02.498020887 +0000 UTC m=+66.168512535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs") pod "network-metrics-daemon-gb86p" (UID: "585a203a-d420-4b43-b6f4-248884bede2b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:30.599130 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.599099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:30.599287 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.599170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:30.599287 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599280    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:30.599402 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599286    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:30.599402 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599307    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:30.599402 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599318    2555 projected.go:196] Error preparing data for projected volume kube-api-access-l64rl for pod openshift-network-diagnostics/network-check-target-d9xcx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:30.599402 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599343    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret podName:639af4e4-909c-4f58-b7ca-f1c2ba762f03 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:02.599324506 +0000 UTC m=+66.269816149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret") pod "global-pull-secret-syncer-wffr5" (UID: "639af4e4-909c-4f58-b7ca-f1c2ba762f03") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:30.599402 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.599379    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl podName:2546be35-ea90-4047-8bdd-8c1f30697226 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:02.599368625 +0000 UTC m=+66.269860258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l64rl" (UniqueName: "kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl") pod "network-check-target-d9xcx" (UID: "2546be35-ea90-4047-8bdd-8c1f30697226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:30.856340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.856309    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:30.856505 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.856352    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:30.856505 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.856451    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:30.856619 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:30.856532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:30.856619 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.856521    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:30.856619 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:30.856605    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:32.855070 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:32.855040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:32.855499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:32.855079    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:32.855499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:32.855142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:32.855499 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:32.855159    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:32.855499 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:32.855210    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:32.855499 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:32.855266    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:33.045930 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:33.045904    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="7cb4f56072f2fb87e9ab743d73b560696b27ded8da0652c601763fa4b8c519af" exitCode=0
Jul 27 21:04:33.046060 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:33.045957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"7cb4f56072f2fb87e9ab743d73b560696b27ded8da0652c601763fa4b8c519af"}
Jul 27 21:04:34.053624 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:34.053586    2555 generic.go:356] "Generic (PLEG): container finished" podID="c18b8d6e-6d55-4925-affb-96030d7814e3" containerID="02861d653bad8400ec8e2cb1f8bd8d09c3fbeec1aa39c5ad1e886ca2a2d9bc49" exitCode=0
Jul 27 21:04:34.054163 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:34.053647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerDied","Data":"02861d653bad8400ec8e2cb1f8bd8d09c3fbeec1aa39c5ad1e886ca2a2d9bc49"}
Jul 27 21:04:34.852499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:34.852471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:34.852664 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:34.852472    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:34.852736 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:34.852481    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:34.852817 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:34.852570    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:34.852817 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:34.852665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:34.852817 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:34.852766    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:35.059988 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:35.059946    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" event={"ID":"c18b8d6e-6d55-4925-affb-96030d7814e3","Type":"ContainerStarted","Data":"32d5e709497414d9b44d626bdf01d3b4abc7d27c42ec392c52f68822494cfbcf"}
Jul 27 21:04:35.085399 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:35.085355    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2c5z6" podStartSLOduration=10.968918034 podStartE2EDuration="39.085340621s" totalImagesPullingTime="28.116422587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:35.085273565 +0000 UTC m=+38.755765217" watchObservedRunningTime="2026-07-27 21:04:35.085340621 +0000 UTC m=+38.755832269"
Jul 27 21:04:36.853535 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:36.853499    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:36.853919 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:36.853583    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:36.853919 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:36.853629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:36.853919 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:36.853676    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:36.853919 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:36.853705    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:36.853919 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:36.853746    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:38.852996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:38.852959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:38.852996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:38.852998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:38.853413 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:38.852966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:38.853413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:38.853072    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:38.853413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:38.853160    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:38.853413 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:38.853239    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:40.852544 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:40.852515    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:40.852544 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:40.852535    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:40.853140 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:40.852542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:40.853140 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:40.852604    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:40.853140 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:40.852673    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:40.853283 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:40.853200    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:42.852493 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:42.852460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:42.852493 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:42.852475    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:42.852941 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:42.852460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:42.852941 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:42.852567    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xcx" podUID="2546be35-ea90-4047-8bdd-8c1f30697226"
Jul 27 21:04:42.852941 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:42.852665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gb86p" podUID="585a203a-d420-4b43-b6f4-248884bede2b"
Jul 27 21:04:42.852941 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:42.852738    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wffr5" podUID="639af4e4-909c-4f58-b7ca-f1c2ba762f03"
Jul 27 21:04:44.605017 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.604989    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-121.ec2.internal" event="NodeReady"
Jul 27 21:04:44.605395 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.605097    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:04:44.676220 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.676186    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w6plj"]
Jul 27 21:04:44.692556 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.692526    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6np85"]
Jul 27 21:04:44.692698 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.692638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.695899 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.695821    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:04:44.696058 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.695908    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:04:44.696142 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.696135    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jkr4s\""
Jul 27 21:04:44.698539 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.698517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42f55a8-25f7-4a39-887a-aaf55094d967-tmp-dir\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.698642 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.698568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b42f55a8-25f7-4a39-887a-aaf55094d967-metrics-tls\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.698642 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.698619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl86m\" (UniqueName: \"kubernetes.io/projected/b42f55a8-25f7-4a39-887a-aaf55094d967-kube-api-access-hl86m\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.698739 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.698655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f55a8-25f7-4a39-887a-aaf55094d967-config-volume\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.708614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.708595    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6plj"]
Jul 27 21:04:44.708698 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.708683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.710913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.710892    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6np85"]
Jul 27 21:04:44.711809 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.711791    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:04:44.711905 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.711831    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:04:44.711975 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.711905    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5dc2\""
Jul 27 21:04:44.712148 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.712133    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:04:44.712297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.712284    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:04:44.770041 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.770003    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g58pk"]
Jul 27 21:04:44.792613 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.792585    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g58pk"]
Jul 27 21:04:44.792785 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.792679    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.796822 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.796799    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:04:44.796957 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.796837    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:04:44.797122 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.797104    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xwcxr\""
Jul 27 21:04:44.797333 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.797317    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:04:44.798946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.798920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cabefb-b80f-42cf-8512-8635090b8873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.799036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.798963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cabefb-b80f-42cf-8512-8635090b8873-crio-socket\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.799036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.798981    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cabefb-b80f-42cf-8512-8635090b8873-data-volume\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.799036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42f55a8-25f7-4a39-887a-aaf55094d967-tmp-dir\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.799148 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b42f55a8-25f7-4a39-887a-aaf55094d967-metrics-tls\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.799192 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799144    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl86m\" (UniqueName: \"kubernetes.io/projected/b42f55a8-25f7-4a39-887a-aaf55094d967-kube-api-access-hl86m\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.799192 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799175    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cabefb-b80f-42cf-8512-8635090b8873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.799269 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799201    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-cert\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.799269 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnnw\" (UniqueName: \"kubernetes.io/projected/77cabefb-b80f-42cf-8512-8635090b8873-kube-api-access-tdnnw\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.799269 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f55a8-25f7-4a39-887a-aaf55094d967-config-volume\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.799409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrvh\" (UniqueName: \"kubernetes.io/projected/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-kube-api-access-dmrvh\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.799409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42f55a8-25f7-4a39-887a-aaf55094d967-tmp-dir\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.799822 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.799801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f55a8-25f7-4a39-887a-aaf55094d967-config-volume\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.803017 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.802997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b42f55a8-25f7-4a39-887a-aaf55094d967-metrics-tls\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.809666 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.809647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl86m\" (UniqueName: \"kubernetes.io/projected/b42f55a8-25f7-4a39-887a-aaf55094d967-kube-api-access-hl86m\") pod \"dns-default-w6plj\" (UID: \"b42f55a8-25f7-4a39-887a-aaf55094d967\") " pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:44.852977 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.852946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:04:44.853167 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.852946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:04:44.853167 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.853091    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:04:44.856060 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.856003    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\""
Jul 27 21:04:44.856377 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.856355    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:04:44.856519 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.856435    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:04:44.856574 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.856518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\""
Jul 27 21:04:44.856653 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.856637    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:04:44.857051 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.857036    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:04:44.900213 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cabefb-b80f-42cf-8512-8635090b8873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900213 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900213    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-cert\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.900406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnnw\" (UniqueName: \"kubernetes.io/projected/77cabefb-b80f-42cf-8512-8635090b8873-kube-api-access-tdnnw\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrvh\" (UniqueName: \"kubernetes.io/projected/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-kube-api-access-dmrvh\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.900406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cabefb-b80f-42cf-8512-8635090b8873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cabefb-b80f-42cf-8512-8635090b8873-crio-socket\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cabefb-b80f-42cf-8512-8635090b8873-data-volume\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900624 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cabefb-b80f-42cf-8512-8635090b8873-crio-socket\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.900794 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.900768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cabefb-b80f-42cf-8512-8635090b8873-data-volume\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.901033 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.901012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cabefb-b80f-42cf-8512-8635090b8873-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.902462 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.902439    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cabefb-b80f-42cf-8512-8635090b8873-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.902609 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.902593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-cert\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:44.908790 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.908746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnnw\" (UniqueName: \"kubernetes.io/projected/77cabefb-b80f-42cf-8512-8635090b8873-kube-api-access-tdnnw\") pod \"insights-runtime-extractor-6np85\" (UID: \"77cabefb-b80f-42cf-8512-8635090b8873\") " pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:44.909023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:44.909007    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrvh\" (UniqueName: \"kubernetes.io/projected/e842bd78-9c4e-47fe-b057-a3ab1db8d36b-kube-api-access-dmrvh\") pod \"ingress-canary-g58pk\" (UID: \"e842bd78-9c4e-47fe-b057-a3ab1db8d36b\") " pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:45.003117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.003027    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:45.017183 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.017159    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6np85"
Jul 27 21:04:45.101563 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.101531    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g58pk"
Jul 27 21:04:45.230655 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.230624    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6plj"]
Jul 27 21:04:45.234676 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:04:45.234644    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42f55a8_25f7_4a39_887a_aaf55094d967.slice/crio-4fd8aebb7d559863f9f3fde91721932ec844af2e29d811f79c1176e6d4fffa5c WatchSource:0}: Error finding container 4fd8aebb7d559863f9f3fde91721932ec844af2e29d811f79c1176e6d4fffa5c: Status 404 returned error can't find the container with id 4fd8aebb7d559863f9f3fde91721932ec844af2e29d811f79c1176e6d4fffa5c
Jul 27 21:04:45.244051 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.244021    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6np85"]
Jul 27 21:04:45.258049 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:04:45.258024    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cabefb_b80f_42cf_8512_8635090b8873.slice/crio-48d560c62ba3e7963a68832a598291ff033ff19f8bdad5a73eaf5aef9b2d1712 WatchSource:0}: Error finding container 48d560c62ba3e7963a68832a598291ff033ff19f8bdad5a73eaf5aef9b2d1712: Status 404 returned error can't find the container with id 48d560c62ba3e7963a68832a598291ff033ff19f8bdad5a73eaf5aef9b2d1712
Jul 27 21:04:45.286044 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:45.286015    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g58pk"]
Jul 27 21:04:45.288892 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:04:45.288867    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode842bd78_9c4e_47fe_b057_a3ab1db8d36b.slice/crio-1d1a65563cb480136b96ac7eeb9ac4d34eb44d8c76ec6b54edcff2211bdb25a9 WatchSource:0}: Error finding container 1d1a65563cb480136b96ac7eeb9ac4d34eb44d8c76ec6b54edcff2211bdb25a9: Status 404 returned error can't find the container with id 1d1a65563cb480136b96ac7eeb9ac4d34eb44d8c76ec6b54edcff2211bdb25a9
Jul 27 21:04:46.088465 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:46.088413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6plj" event={"ID":"b42f55a8-25f7-4a39-887a-aaf55094d967","Type":"ContainerStarted","Data":"4fd8aebb7d559863f9f3fde91721932ec844af2e29d811f79c1176e6d4fffa5c"}
Jul 27 21:04:46.090534 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:46.090504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g58pk" event={"ID":"e842bd78-9c4e-47fe-b057-a3ab1db8d36b","Type":"ContainerStarted","Data":"1d1a65563cb480136b96ac7eeb9ac4d34eb44d8c76ec6b54edcff2211bdb25a9"}
Jul 27 21:04:46.093113 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:46.093077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6np85" event={"ID":"77cabefb-b80f-42cf-8512-8635090b8873","Type":"ContainerStarted","Data":"48d560c62ba3e7963a68832a598291ff033ff19f8bdad5a73eaf5aef9b2d1712"}
Jul 27 21:04:49.360315 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.360280    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"]
Jul 27 21:04:49.477042 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.477003    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"]
Jul 27 21:04:49.477222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.477148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:49.480674 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.480641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vgmwc\""
Jul 27 21:04:49.481572 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.481512    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 21:04:49.550764 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.550700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-79w6s\" (UID: \"e726fa5f-8c90-453a-95c4-6571d9ae2756\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:49.652065 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:49.651978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-79w6s\" (UID: \"e726fa5f-8c90-453a-95c4-6571d9ae2756\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:49.652208 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:49.652118    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:04:49.652208 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:49.652186    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates podName:e726fa5f-8c90-453a-95c4-6571d9ae2756 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:50.152169559 +0000 UTC m=+53.822661185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-79w6s" (UID: "e726fa5f-8c90-453a-95c4-6571d9ae2756") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:04:50.154880 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:50.154827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-79w6s\" (UID: \"e726fa5f-8c90-453a-95c4-6571d9ae2756\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:50.155036 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:50.154989    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:04:50.155154 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:50.155068    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates podName:e726fa5f-8c90-453a-95c4-6571d9ae2756 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:51.155048418 +0000 UTC m=+54.825540067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-79w6s" (UID: "e726fa5f-8c90-453a-95c4-6571d9ae2756") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:04:51.163590 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:51.163497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-79w6s\" (UID: \"e726fa5f-8c90-453a-95c4-6571d9ae2756\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:51.166463 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:51.166431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e726fa5f-8c90-453a-95c4-6571d9ae2756-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-79w6s\" (UID: \"e726fa5f-8c90-453a-95c4-6571d9ae2756\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:51.287653 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:51.287614    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:51.792779 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:51.791435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"]
Jul 27 21:04:52.112426 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.112381    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6np85" event={"ID":"77cabefb-b80f-42cf-8512-8635090b8873","Type":"ContainerStarted","Data":"da29763772d28b3aea8a5ad04b8bd58c361a7adc57712e02b8086f1d6d468c43"}
Jul 27 21:04:52.114614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.114584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s" event={"ID":"e726fa5f-8c90-453a-95c4-6571d9ae2756","Type":"ContainerStarted","Data":"7153c7e1f355af67262cbb3dce6ecfe47f0297efa7e95b7e3c6602d18aa7bbb6"}
Jul 27 21:04:52.116378 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.116346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g58pk" event={"ID":"e842bd78-9c4e-47fe-b057-a3ab1db8d36b","Type":"ContainerStarted","Data":"f38a354a7b05271fee724ae2a0b9f9365eed218d10a1b2515051979bad9d3899"}
Jul 27 21:04:52.119044 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.119010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6plj" event={"ID":"b42f55a8-25f7-4a39-887a-aaf55094d967","Type":"ContainerStarted","Data":"1b4e5440a6e0d8f907d1844f3665f9c0029d9470ee93f1818d65187194bcb6fc"}
Jul 27 21:04:52.119163 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.119066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6plj" event={"ID":"b42f55a8-25f7-4a39-887a-aaf55094d967","Type":"ContainerStarted","Data":"9be3b2678c959c7129958bdb67a8d73c649008c7ac6aaa29e239b504a8de3d44"}
Jul 27 21:04:52.119350 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.119320    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w6plj"
Jul 27 21:04:52.136571 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.136519    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g58pk" podStartSLOduration=1.772304572 podStartE2EDuration="8.136502355s" totalImagesPullingTime="6.364197783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:52.136159123 +0000 UTC m=+55.806650772" watchObservedRunningTime="2026-07-27 21:04:52.136502355 +0000 UTC m=+55.806994003"
Jul 27 21:04:52.155778 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:52.155694    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w6plj" podStartSLOduration=1.7369129920000002 podStartE2EDuration="8.155672014s" totalImagesPullingTime="6.418759022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:52.154907218 +0000 UTC m=+55.825398868" watchObservedRunningTime="2026-07-27 21:04:52.155672014 +0000 UTC m=+55.826163665"
Jul 27 21:04:54.126769 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.126728    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s" event={"ID":"e726fa5f-8c90-453a-95c4-6571d9ae2756","Type":"ContainerStarted","Data":"66ad7e5d415b0fd5f0c1e0cbbe09934c32597e41fdff7e1a5fb159dc04dae801"}
Jul 27 21:04:54.127234 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.126959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:54.132133 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.132095    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s"
Jul 27 21:04:54.144572 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.144513    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-79w6s" podStartSLOduration=3.848766077 podStartE2EDuration="5.144492062s" totalImagesPullingTime="1.295725985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.143643188 +0000 UTC m=+57.814134836" watchObservedRunningTime="2026-07-27 21:04:54.144492062 +0000 UTC m=+57.814983711"
Jul 27 21:04:54.443638 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.443561    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-srpsp"]
Jul 27 21:04:54.454514 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.454484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.457496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.457467    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-srpsp"]
Jul 27 21:04:54.457627 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.457507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 21:04:54.457793 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.457775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-68j42\""
Jul 27 21:04:54.457883 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.457777    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:04:54.459007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.458865    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:04:54.459007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.458892    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:04:54.459007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.458900    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 21:04:54.491300 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.491268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26fedb55-d188-40bf-a07d-9074c0a597db-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.491300 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.491302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltzv\" (UniqueName: \"kubernetes.io/projected/26fedb55-d188-40bf-a07d-9074c0a597db-kube-api-access-9ltzv\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.491496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.491332    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.491496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.491410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.592154 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.592112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.592347 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.592205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26fedb55-d188-40bf-a07d-9074c0a597db-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.592347 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.592233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltzv\" (UniqueName: \"kubernetes.io/projected/26fedb55-d188-40bf-a07d-9074c0a597db-kube-api-access-9ltzv\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.592347 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.592273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.592347 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:54.592284    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 21:04:54.592576 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:04:54.592362    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls podName:26fedb55-d188-40bf-a07d-9074c0a597db nodeName:}" failed. No retries permitted until 2026-07-27 21:04:55.09234156 +0000 UTC m=+58.762833189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-srpsp" (UID: "26fedb55-d188-40bf-a07d-9074c0a597db") : secret "prometheus-operator-tls" not found
Jul 27 21:04:54.593071 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.593038    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26fedb55-d188-40bf-a07d-9074c0a597db-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.594925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.594905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:54.603975 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:54.603950    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltzv\" (UniqueName: \"kubernetes.io/projected/26fedb55-d188-40bf-a07d-9074c0a597db-kube-api-access-9ltzv\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:55.097170 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:55.097116    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:55.099687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:55.099660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/26fedb55-d188-40bf-a07d-9074c0a597db-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-srpsp\" (UID: \"26fedb55-d188-40bf-a07d-9074c0a597db\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:55.364301 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:55.364203    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp"
Jul 27 21:04:55.486246 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:55.486216    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-srpsp"]
Jul 27 21:04:55.490566 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:04:55.490533    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fedb55_d188_40bf_a07d_9074c0a597db.slice/crio-cdcbcc39a8c7f1df455f221d8d2fe2ac5728dec4b4ebe79da15f182dbd420ad5 WatchSource:0}: Error finding container cdcbcc39a8c7f1df455f221d8d2fe2ac5728dec4b4ebe79da15f182dbd420ad5: Status 404 returned error can't find the container with id cdcbcc39a8c7f1df455f221d8d2fe2ac5728dec4b4ebe79da15f182dbd420ad5
Jul 27 21:04:56.034024 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:56.033987    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926vh"
Jul 27 21:04:56.134168 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:56.134131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp" event={"ID":"26fedb55-d188-40bf-a07d-9074c0a597db","Type":"ContainerStarted","Data":"cdcbcc39a8c7f1df455f221d8d2fe2ac5728dec4b4ebe79da15f182dbd420ad5"}
Jul 27 21:04:56.136259 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:56.136224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6np85" event={"ID":"77cabefb-b80f-42cf-8512-8635090b8873","Type":"ContainerStarted","Data":"eff49ecd15e948f3b8396ebbdfdda73ee60f8f570785a8bb7318cc8bb36f2612"}
Jul 27 21:04:59.146309 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:59.146277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp" event={"ID":"26fedb55-d188-40bf-a07d-9074c0a597db","Type":"ContainerStarted","Data":"46080beecf84cd83086bdf883663c99380e0884bedac1999905fc0122f2ad243"}
Jul 27 21:04:59.148440 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:59.148413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6np85" event={"ID":"77cabefb-b80f-42cf-8512-8635090b8873","Type":"ContainerStarted","Data":"f38eccbfd8f14d67a87bbf2b121283b0f9c08c31272b9a354915cb39a40f6a27"}
Jul 27 21:04:59.184043 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:04:59.183536    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6np85" podStartSLOduration=1.524499614 podStartE2EDuration="15.183516253s" totalImagesPullingTime="13.659016639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.181303275 +0000 UTC m=+62.851794923" watchObservedRunningTime="2026-07-27 21:04:59.183516253 +0000 UTC m=+62.854007902"
Jul 27 21:05:00.153415 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:00.153374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp" event={"ID":"26fedb55-d188-40bf-a07d-9074c0a597db","Type":"ContainerStarted","Data":"05ff74f5fb0eb683773b547f7dda40492c2f612132df313b57aa341380da2bae"}
Jul 27 21:05:00.173440 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:00.173395    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-srpsp" podStartSLOduration=2.63575312 podStartE2EDuration="6.173380398s" totalImagesPullingTime="3.537627278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:00.172985461 +0000 UTC m=+63.843477111" watchObservedRunningTime="2026-07-27 21:05:00.173380398 +0000 UTC m=+63.843872046"
Jul 27 21:05:01.808243 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.808207    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"]
Jul 27 21:05:01.810280 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.810259    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.813006 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.812979    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 21:05:01.813138 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.812986    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-28lgc\""
Jul 27 21:05:01.813138 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.813123    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:05:01.829703 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.829673    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"]
Jul 27 21:05:01.832376 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.832347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"]
Jul 27 21:05:01.832506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.832398    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t9t54"]
Jul 27 21:05:01.832506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.832456    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.834800 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.834782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.837870 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.837849    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:05:01.838023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.837873    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 21:05:01.838023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.837876    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:05:01.838194 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.837884    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:05:01.838194 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.837930    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xcjgp\""
Jul 27 21:05:01.838294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.838240    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ntxzb\""
Jul 27 21:05:01.838713 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.838694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 21:05:01.838713 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.838705    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:05:01.848554 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.848530    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"]
Jul 27 21:05:01.853279 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.853256    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.853419 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.853285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6dc33-1654-44e2-aa4e-3425a6003376-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.853419 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.853368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhh6\" (UniqueName: \"kubernetes.io/projected/26c6dc33-1654-44e2-aa4e-3425a6003376-kube-api-access-vjhh6\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.853487 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.853435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.953784 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953830    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhh6\" (UniqueName: \"kubernetes.io/projected/26c6dc33-1654-44e2-aa4e-3425a6003376-kube-api-access-vjhh6\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wg9\" (UniqueName: \"kubernetes.io/projected/565e4840-63af-4c39-94a1-677767f34256-kube-api-access-86wg9\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/565e4840-63af-4c39-94a1-677767f34256-node-exporter-textfile\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.953984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-tls\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.953998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:01.954066    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-node-exporter-wtmp\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6dc33-1654-44e2-aa4e-3425a6003376-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:01.954136    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls podName:26c6dc33-1654-44e2-aa4e-3425a6003376 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:02.454119641 +0000 UTC m=+66.124611267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-77tmg" (UID: "26c6dc33-1654-44e2-aa4e-3425a6003376") : secret "openshift-state-metrics-tls" not found
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954199    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-node-exporter-accelerators-collector-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkldd\" (UniqueName: \"kubernetes.io/projected/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-api-access-lkldd\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.954325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.954680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954363    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:01.954680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-root\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-metrics-client-ca\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954459    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-sys\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:01.954839 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.954681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26c6dc33-1654-44e2-aa4e-3425a6003376-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.956391 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.956365    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:01.962932 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:01.962909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhh6\" (UniqueName: \"kubernetes.io/projected/26c6dc33-1654-44e2-aa4e-3425a6003376-kube-api-access-vjhh6\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:02.055247 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-tls\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055247 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-node-exporter-wtmp\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-node-exporter-accelerators-collector-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkldd\" (UniqueName: \"kubernetes.io/projected/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-api-access-lkldd\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-root\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-metrics-client-ca\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055523    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-sys\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-node-exporter-wtmp\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wg9\" (UniqueName: \"kubernetes.io/projected/565e4840-63af-4c39-94a1-677767f34256-kube-api-access-86wg9\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.055973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.055836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/565e4840-63af-4c39-94a1-677767f34256-node-exporter-textfile\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-metrics-client-ca\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/565e4840-63af-4c39-94a1-677767f34256-node-exporter-textfile\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056152    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/565e4840-63af-4c39-94a1-677767f34256-node-exporter-accelerators-collector-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-sys\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056265 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:02.056240    2555 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 27 21:05:02.056561 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:02.056299    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls podName:4f0cb58b-ec9e-4825-a6d0-aae37b76e273 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:02.556282056 +0000 UTC m=+66.226773706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-fsfdn" (UID: "4f0cb58b-ec9e-4825-a6d0-aae37b76e273") : secret "kube-state-metrics-tls" not found
Jul 27 21:05:02.056561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/565e4840-63af-4c39-94a1-677767f34256-root\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.056561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.056561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.056519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.057941 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.057920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.058034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.057921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-tls\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.058275 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.058236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/565e4840-63af-4c39-94a1-677767f34256-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.066721 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.066699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wg9\" (UniqueName: \"kubernetes.io/projected/565e4840-63af-4c39-94a1-677767f34256-kube-api-access-86wg9\") pod \"node-exporter-t9t54\" (UID: \"565e4840-63af-4c39-94a1-677767f34256\") " pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.066841 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.066745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkldd\" (UniqueName: \"kubernetes.io/projected/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-api-access-lkldd\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.124220 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.124189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w6plj"
Jul 27 21:05:02.147808 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.147779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t9t54"
Jul 27 21:05:02.158365 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:02.158339    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565e4840_63af_4c39_94a1_677767f34256.slice/crio-4d0b41f31db2d8a08a953162f924c7d228488da20fe784fbcbd8892a9533db62 WatchSource:0}: Error finding container 4d0b41f31db2d8a08a953162f924c7d228488da20fe784fbcbd8892a9533db62: Status 404 returned error can't find the container with id 4d0b41f31db2d8a08a953162f924c7d228488da20fe784fbcbd8892a9533db62
Jul 27 21:05:02.163796 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.163770    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t9t54" event={"ID":"565e4840-63af-4c39-94a1-677767f34256","Type":"ContainerStarted","Data":"4d0b41f31db2d8a08a953162f924c7d228488da20fe784fbcbd8892a9533db62"}
Jul 27 21:05:02.459505 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.459467    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:02.459791 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:02.459614    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 21:05:02.459791 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:02.459678    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls podName:26c6dc33-1654-44e2-aa4e-3425a6003376 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:03.459663264 +0000 UTC m=+67.130154894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-77tmg" (UID: "26c6dc33-1654-44e2-aa4e-3425a6003376") : secret "openshift-state-metrics-tls" not found
Jul 27 21:05:02.560255 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.560220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.560426 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.560282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:05:02.562596 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.562566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f0cb58b-ec9e-4825-a6d0-aae37b76e273-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fsfdn\" (UID: \"4f0cb58b-ec9e-4825-a6d0-aae37b76e273\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.563048 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.563032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:02.572570 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.572545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585a203a-d420-4b43-b6f4-248884bede2b-metrics-certs\") pod \"network-metrics-daemon-gb86p\" (UID: \"585a203a-d420-4b43-b6f4-248884bede2b\") " pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:05:02.576084 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.576065    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\""
Jul 27 21:05:02.583618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.583602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gb86p"
Jul 27 21:05:02.661037 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.661001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:05:02.661186 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.661070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:05:02.664266 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.664059    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:02.664266 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.664065    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:02.674650 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.674370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/639af4e4-909c-4f58-b7ca-f1c2ba762f03-original-pull-secret\") pod \"global-pull-secret-syncer-wffr5\" (UID: \"639af4e4-909c-4f58-b7ca-f1c2ba762f03\") " pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:05:02.674650 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.674459    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:02.684547 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.684518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64rl\" (UniqueName: \"kubernetes.io/projected/2546be35-ea90-4047-8bdd-8c1f30697226-kube-api-access-l64rl\") pod \"network-check-target-d9xcx\" (UID: \"2546be35-ea90-4047-8bdd-8c1f30697226\") " pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:05:02.720522 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.720428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gb86p"]
Jul 27 21:05:02.723220 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:02.723194    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585a203a_d420_4b43_b6f4_248884bede2b.slice/crio-4a281ddcd1bee7224550e125cf81ad251db70544af982c73934c4b3f13d827d5 WatchSource:0}: Error finding container 4a281ddcd1bee7224550e125cf81ad251db70544af982c73934c4b3f13d827d5: Status 404 returned error can't find the container with id 4a281ddcd1bee7224550e125cf81ad251db70544af982c73934c4b3f13d827d5
Jul 27 21:05:02.742459 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.742436    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"
Jul 27 21:05:02.865948 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.865919    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\""
Jul 27 21:05:02.869052 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.869021    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wffr5"
Jul 27 21:05:02.873261 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.873213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:05:02.891996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.891969    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:05:02.896571 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.896542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.900405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.900271    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:05:02.901896 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.901825    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:05:02.902333 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.902313    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:05:02.902663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.902636    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mldfb\""
Jul 27 21:05:02.902901 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.902884    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:05:02.903228 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.903209    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:05:02.904437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.903485    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:05:02.904437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.903649    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:05:02.904437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.903843    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:05:02.904437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.904036    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn"]
Jul 27 21:05:02.904437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.904085    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:05:02.917409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.917074    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:05:02.964637 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964553    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.964809 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964637    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfl9t\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.964809 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964693    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.964809 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964729    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.964809 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.964984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965239 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.965041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965239 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.965092    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965239 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.965132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:02.965239 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:02.965168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.027264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.027219    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d9xcx"]
Jul 27 21:05:03.046193 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.046161    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wffr5"]
Jul 27 21:05:03.066002 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.065969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066286 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066286 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066286 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066286 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:05:03.066261    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle podName:4a309ec8-3d07-4715-bf39-edb8bb31f494 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:03.566236502 +0000 UTC m=+67.236728152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494") : configmap references non-existent config key: ca-bundle.crt
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfl9t\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.066481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.066452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.067384 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.067357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.067923 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.067625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.068915 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.068870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.069266 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.069238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.069461 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.069437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.070730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.070686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.071385 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.071369    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.071536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.071421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.071606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.071533    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.071986 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.071948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.072920 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.072901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.076547 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.076527    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfl9t\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.150426 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:03.150394    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2546be35_ea90_4047_8bdd_8c1f30697226.slice/crio-ae79c3c531dbc520cea46556372873da4d3d3c38747472eee3f80cd1a51480e4 WatchSource:0}: Error finding container ae79c3c531dbc520cea46556372873da4d3d3c38747472eee3f80cd1a51480e4: Status 404 returned error can't find the container with id ae79c3c531dbc520cea46556372873da4d3d3c38747472eee3f80cd1a51480e4
Jul 27 21:05:03.151121 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:03.151100    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639af4e4_909c_4f58_b7ca_f1c2ba762f03.slice/crio-532785f5669dc3b3f888be27beb5b3aee4e227d20f76db1102710b3ccdde54a5 WatchSource:0}: Error finding container 532785f5669dc3b3f888be27beb5b3aee4e227d20f76db1102710b3ccdde54a5: Status 404 returned error can't find the container with id 532785f5669dc3b3f888be27beb5b3aee4e227d20f76db1102710b3ccdde54a5
Jul 27 21:05:03.151965 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:03.151938    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0cb58b_ec9e_4825_a6d0_aae37b76e273.slice/crio-1d0278f11732671d77b7ce2ac94b9b340261dcd37d6bbe7a79bff91bb7d6dcb8 WatchSource:0}: Error finding container 1d0278f11732671d77b7ce2ac94b9b340261dcd37d6bbe7a79bff91bb7d6dcb8: Status 404 returned error can't find the container with id 1d0278f11732671d77b7ce2ac94b9b340261dcd37d6bbe7a79bff91bb7d6dcb8
Jul 27 21:05:03.168414 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.168382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wffr5" event={"ID":"639af4e4-909c-4f58-b7ca-f1c2ba762f03","Type":"ContainerStarted","Data":"532785f5669dc3b3f888be27beb5b3aee4e227d20f76db1102710b3ccdde54a5"}
Jul 27 21:05:03.169635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.169603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn" event={"ID":"4f0cb58b-ec9e-4825-a6d0-aae37b76e273","Type":"ContainerStarted","Data":"1d0278f11732671d77b7ce2ac94b9b340261dcd37d6bbe7a79bff91bb7d6dcb8"}
Jul 27 21:05:03.170720 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.170681    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d9xcx" event={"ID":"2546be35-ea90-4047-8bdd-8c1f30697226","Type":"ContainerStarted","Data":"ae79c3c531dbc520cea46556372873da4d3d3c38747472eee3f80cd1a51480e4"}
Jul 27 21:05:03.172087 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.172066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gb86p" event={"ID":"585a203a-d420-4b43-b6f4-248884bede2b","Type":"ContainerStarted","Data":"4a281ddcd1bee7224550e125cf81ad251db70544af982c73934c4b3f13d827d5"}
Jul 27 21:05:03.469522 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.469485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:03.472323 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.472298    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26c6dc33-1654-44e2-aa4e-3425a6003376-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-77tmg\" (UID: \"26c6dc33-1654-44e2-aa4e-3425a6003376\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:03.570672 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.570644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.571891 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.571859    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.619435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.619401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"
Jul 27 21:05:03.793690 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.793353    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f94c686b7-9fk92"]
Jul 27 21:05:03.800884 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.800860    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.803649 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.803621    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.803993    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.804026    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.804059    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vkwwr\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.804108    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-92ggg8ql0sc2g\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.804110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 21:05:03.804162 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.804160    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 21:05:03.808328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.808302    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f94c686b7-9fk92"]
Jul 27 21:05:03.817893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.817868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873081    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873171    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-grpc-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873197    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52jr\" (UniqueName: \"kubernetes.io/projected/7e60d566-2700-488e-aedd-56107c1a1c4a-kube-api-access-l52jr\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873786 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e60d566-2700-488e-aedd-56107c1a1c4a-metrics-client-ca\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.873786 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.873260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974470 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974617    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-grpc-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974881 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52jr\" (UniqueName: \"kubernetes.io/projected/7e60d566-2700-488e-aedd-56107c1a1c4a-kube-api-access-l52jr\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974881 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e60d566-2700-488e-aedd-56107c1a1c4a-metrics-client-ca\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.974881 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.974714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.976691 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.976515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e60d566-2700-488e-aedd-56107c1a1c4a-metrics-client-ca\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.978744 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.978674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.979983 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.979954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.980148 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.980121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.980948 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.980925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.981794 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.981768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.982235 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.982217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e60d566-2700-488e-aedd-56107c1a1c4a-secret-grpc-tls\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.988878 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.988843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52jr\" (UniqueName: \"kubernetes.io/projected/7e60d566-2700-488e-aedd-56107c1a1c4a-kube-api-access-l52jr\") pod \"thanos-querier-5f94c686b7-9fk92\" (UID: \"7e60d566-2700-488e-aedd-56107c1a1c4a\") " pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:03.997007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:03.996975    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg"]
Jul 27 21:05:04.026564 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.026530    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:05:04.113671 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.113583    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:04.114480 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:04.114449    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c6dc33_1654_44e2_aa4e_3425a6003376.slice/crio-fd6c27f63b2132aa7de21271024ad72eeabc503d972712101254bd4c6ea909d1 WatchSource:0}: Error finding container fd6c27f63b2132aa7de21271024ad72eeabc503d972712101254bd4c6ea909d1: Status 404 returned error can't find the container with id fd6c27f63b2132aa7de21271024ad72eeabc503d972712101254bd4c6ea909d1
Jul 27 21:05:04.115225 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:04.115201    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a309ec8_3d07_4715_bf39_edb8bb31f494.slice/crio-fe313f0ae15dbb0cb76d75360ec8277b2a4bb8d2b5bb2727bf103a6356599f8e WatchSource:0}: Error finding container fe313f0ae15dbb0cb76d75360ec8277b2a4bb8d2b5bb2727bf103a6356599f8e: Status 404 returned error can't find the container with id fe313f0ae15dbb0cb76d75360ec8277b2a4bb8d2b5bb2727bf103a6356599f8e
Jul 27 21:05:04.178373 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.178337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"fe313f0ae15dbb0cb76d75360ec8277b2a4bb8d2b5bb2727bf103a6356599f8e"}
Jul 27 21:05:04.180296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.180195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg" event={"ID":"26c6dc33-1654-44e2-aa4e-3425a6003376","Type":"ContainerStarted","Data":"fd6c27f63b2132aa7de21271024ad72eeabc503d972712101254bd4c6ea909d1"}
Jul 27 21:05:04.182151 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.182016    2555 generic.go:356] "Generic (PLEG): container finished" podID="565e4840-63af-4c39-94a1-677767f34256" containerID="0066f4da558d8d183cf0001c5d51666e1315aeb4a986d34c291729c304d67933" exitCode=0
Jul 27 21:05:04.182151 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.182068    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t9t54" event={"ID":"565e4840-63af-4c39-94a1-677767f34256","Type":"ContainerDied","Data":"0066f4da558d8d183cf0001c5d51666e1315aeb4a986d34c291729c304d67933"}
Jul 27 21:05:04.320619 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:04.320578    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f94c686b7-9fk92"]
Jul 27 21:05:04.327385 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:04.327341    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e60d566_2700_488e_aedd_56107c1a1c4a.slice/crio-68dade265e21fc1357856d6979eecf21dad6243291186985d09492f60d47382f WatchSource:0}: Error finding container 68dade265e21fc1357856d6979eecf21dad6243291186985d09492f60d47382f: Status 404 returned error can't find the container with id 68dade265e21fc1357856d6979eecf21dad6243291186985d09492f60d47382f
Jul 27 21:05:05.188764 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.188648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"68dade265e21fc1357856d6979eecf21dad6243291186985d09492f60d47382f"}
Jul 27 21:05:05.191726 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.191389    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gb86p" event={"ID":"585a203a-d420-4b43-b6f4-248884bede2b","Type":"ContainerStarted","Data":"803b8a87cd6c62fc39903b709b072b3c645e0ff156780ac330818da4395288ae"}
Jul 27 21:05:05.191726 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.191440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gb86p" event={"ID":"585a203a-d420-4b43-b6f4-248884bede2b","Type":"ContainerStarted","Data":"653b114767ac67ab1cbf94770b842addb729ae16ad24a9a8ee81e7f887e564c8"}
Jul 27 21:05:05.194426 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.194396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg" event={"ID":"26c6dc33-1654-44e2-aa4e-3425a6003376","Type":"ContainerStarted","Data":"171f85c8504da37d8f4b798599ee687c2bebec73fc071618bc05caf05d70c051"}
Jul 27 21:05:05.194534 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.194453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg" event={"ID":"26c6dc33-1654-44e2-aa4e-3425a6003376","Type":"ContainerStarted","Data":"00a077407ed084e991426ba2dbafd80025fb23a2530386103aa67f5fcd9d4f5a"}
Jul 27 21:05:05.197579 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.197498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t9t54" event={"ID":"565e4840-63af-4c39-94a1-677767f34256","Type":"ContainerStarted","Data":"b149afca45836cb99b4a33ccc9703194da8e7aac695dd4c1ae82dde5fcb2845d"}
Jul 27 21:05:05.197579 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.197550    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t9t54" event={"ID":"565e4840-63af-4c39-94a1-677767f34256","Type":"ContainerStarted","Data":"c9e0f0b88dc38c2499a74feccc98fbecff9c85ab632f5ca5908ca3e91bc77ad6"}
Jul 27 21:05:05.212546 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.212179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gb86p" podStartSLOduration=67.788810397 podStartE2EDuration="1m9.212159296s" totalImagesPullingTime="1.423348899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:05.210827223 +0000 UTC m=+68.881318872" watchObservedRunningTime="2026-07-27 21:05:05.212159296 +0000 UTC m=+68.882650945"
Jul 27 21:05:05.237829 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:05.237110    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t9t54" podStartSLOduration=3.138733189 podStartE2EDuration="4.237090362s" totalImagesPullingTime="1.034966294s" totalInitContainerRuntime="63.390879ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:05.235535315 +0000 UTC m=+68.906027039" watchObservedRunningTime="2026-07-27 21:05:05.237090362 +0000 UTC m=+68.907582008"
Jul 27 21:05:06.114517 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.114472    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-796887b949-2jbz9"]
Jul 27 21:05:06.120351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.120322    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.124743 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.124713    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 21:05:06.124901 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.124816    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 21:05:06.125318 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.125006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9swdf\""
Jul 27 21:05:06.125318 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.125201    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 21:05:06.125318 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.125229    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e7fh35993vo2s\""
Jul 27 21:05:06.125594 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.125574    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:05:06.131958 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.130659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796887b949-2jbz9"]
Jul 27 21:05:06.192618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b20c43c9-df65-4f26-b8b1-749689361936-audit-log\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.192618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjm77\" (UniqueName: \"kubernetes.io/projected/b20c43c9-df65-4f26-b8b1-749689361936-kube-api-access-hjm77\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.192618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-metrics-server-audit-profiles\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.193196 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-client-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.193196 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-tls\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.193196 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-client-certs\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.193196 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.192901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.294772 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.294601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-tls\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.294772 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.294728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-client-certs\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b20c43c9-df65-4f26-b8b1-749689361936-audit-log\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjm77\" (UniqueName: \"kubernetes.io/projected/b20c43c9-df65-4f26-b8b1-749689361936-kube-api-access-hjm77\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295392    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-metrics-server-audit-profiles\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-client-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.295971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b20c43c9-df65-4f26-b8b1-749689361936-audit-log\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.296225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.296917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.296865    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b20c43c9-df65-4f26-b8b1-749689361936-metrics-server-audit-profiles\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.299485 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.299448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-client-ca-bundle\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.300039 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.299997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-tls\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.301108 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.301085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b20c43c9-df65-4f26-b8b1-749689361936-secret-metrics-server-client-certs\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.308047 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.307997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjm77\" (UniqueName: \"kubernetes.io/projected/b20c43c9-df65-4f26-b8b1-749689361936-kube-api-access-hjm77\") pod \"metrics-server-796887b949-2jbz9\" (UID: \"b20c43c9-df65-4f26-b8b1-749689361936\") " pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.437534 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.437162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:06.567294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.567257    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"]
Jul 27 21:05:06.570080 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.570054    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.574170 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.573950    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 21:05:06.574170 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.573971    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k2gh4\""
Jul 27 21:05:06.580404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.580097    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"]
Jul 27 21:05:06.699082 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.698985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbvl\" (UniqueName: \"kubernetes.io/projected/138fc1a3-ebe3-45bc-a700-5f6f85a88410-kube-api-access-sgbvl\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.699082 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.699045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/138fc1a3-ebe3-45bc-a700-5f6f85a88410-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.799691 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.799544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbvl\" (UniqueName: \"kubernetes.io/projected/138fc1a3-ebe3-45bc-a700-5f6f85a88410-kube-api-access-sgbvl\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.799691 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.799603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/138fc1a3-ebe3-45bc-a700-5f6f85a88410-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.802453 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.802420    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/138fc1a3-ebe3-45bc-a700-5f6f85a88410-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.809210 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.809183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbvl\" (UniqueName: \"kubernetes.io/projected/138fc1a3-ebe3-45bc-a700-5f6f85a88410-kube-api-access-sgbvl\") pod \"monitoring-plugin-68b7fc9574-6mlss\" (UID: \"138fc1a3-ebe3-45bc-a700-5f6f85a88410\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:06.883350 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:06.883314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:08.090896 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.090460    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:05:08.095893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.095862    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.099769 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.099726    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:05:08.099942 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.099909    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:05:08.100023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.099977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:05:08.100095 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.099977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:05:08.100406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.100304    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:05:08.100406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.100386    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:05:08.100567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.100388    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:05:08.100567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.100529    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-47i89gm7bch3\""
Jul 27 21:05:08.101710 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.101689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:05:08.101887 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.101736    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:05:08.101980 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.101738    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:05:08.102182 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.102152    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:05:08.103959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.103634    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:05:08.103959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.103861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xqzbz\""
Jul 27 21:05:08.106403 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.105231    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:05:08.108385 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.108362    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:05:08.115683 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.115656    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:05:08.214325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgjw\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214337    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214513    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214684    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.214893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214840    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214956    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.214976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.215016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215091 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.215061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.215256 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.215092    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316188    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316383    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316439    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316467    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgjw\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316522    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316610    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.317294 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.318262 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.318262 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.318262 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.316872    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.318262 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.318146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.319781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.318818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.319781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.318939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.319781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.319453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.320782 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.320644    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.322126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.320991    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.323723 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.322631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.323723 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.323296    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.323723 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.323402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.323723 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.323591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.324116 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.324088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.325460 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.324646    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.325460 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.325172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.333474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.326121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.333474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.326715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.333474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.327211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.333474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.328476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgjw\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.333474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.330350    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.336183 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.335742    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.336183 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.336164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.411275 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.411179    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:08.740109 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.740073    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796887b949-2jbz9"]
Jul 27 21:05:08.857355 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:08.857323    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20c43c9_df65_4f26_b8b1_749689361936.slice/crio-b802b3688ef44ab9f90586a91a9a686d0eac311ad376b9dcd27539f826305380 WatchSource:0}: Error finding container b802b3688ef44ab9f90586a91a9a686d0eac311ad376b9dcd27539f826305380: Status 404 returned error can't find the container with id b802b3688ef44ab9f90586a91a9a686d0eac311ad376b9dcd27539f826305380
Jul 27 21:05:08.975096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.975063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"]
Jul 27 21:05:08.989370 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:08.989342    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:05:09.214597 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:09.214560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796887b949-2jbz9" event={"ID":"b20c43c9-df65-4f26-b8b1-749689361936","Type":"ContainerStarted","Data":"b802b3688ef44ab9f90586a91a9a686d0eac311ad376b9dcd27539f826305380"}
Jul 27 21:05:09.367491 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:09.367452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138fc1a3_ebe3_45bc_a700_5f6f85a88410.slice/crio-30dddc6231717f2822f73ed339e37535f5f2658e0cd00b9d4bd2d0e5390f892c WatchSource:0}: Error finding container 30dddc6231717f2822f73ed339e37535f5f2658e0cd00b9d4bd2d0e5390f892c: Status 404 returned error can't find the container with id 30dddc6231717f2822f73ed339e37535f5f2658e0cd00b9d4bd2d0e5390f892c
Jul 27 21:05:09.368381 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:05:09.368312    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ad960c_2763_437a_992f_7b7ad9c9aea1.slice/crio-684edf42be841522b3e8303600c2635ae5a731ef33d444a6b6bb2367fe9854b1 WatchSource:0}: Error finding container 684edf42be841522b3e8303600c2635ae5a731ef33d444a6b6bb2367fe9854b1: Status 404 returned error can't find the container with id 684edf42be841522b3e8303600c2635ae5a731ef33d444a6b6bb2367fe9854b1
Jul 27 21:05:10.221333 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.221268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wffr5" event={"ID":"639af4e4-909c-4f58-b7ca-f1c2ba762f03","Type":"ContainerStarted","Data":"86e97365b6db0c0ab504e3b6a3987412b80c02a4899dbee239fc15e6343074bd"}
Jul 27 21:05:10.224605 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.223923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn" event={"ID":"4f0cb58b-ec9e-4825-a6d0-aae37b76e273","Type":"ContainerStarted","Data":"d0887a7e1f428a1df60aeb852b321935e1f19c9252a3fd461274b9cf51a4f2e3"}
Jul 27 21:05:10.224605 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.223985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn" event={"ID":"4f0cb58b-ec9e-4825-a6d0-aae37b76e273","Type":"ContainerStarted","Data":"98b8f67f42ec066588213e3913336ed091207b4c169d21484ccf14b88cd821b2"}
Jul 27 21:05:10.224605 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.224022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn" event={"ID":"4f0cb58b-ec9e-4825-a6d0-aae37b76e273","Type":"ContainerStarted","Data":"75d36643f1452df945f0617ae4f234f3325b39cb83b4129f240e63b18abff25f"}
Jul 27 21:05:10.226648 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.226617    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60" exitCode=0
Jul 27 21:05:10.226785 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.226641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"}
Jul 27 21:05:10.229087 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.228592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d9xcx" event={"ID":"2546be35-ea90-4047-8bdd-8c1f30697226","Type":"ContainerStarted","Data":"b1d81b41dcbbf990c9b56d82fcb724135535b5d0802fb5bebebc903c3513a26d"}
Jul 27 21:05:10.229087 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.228650    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:05:10.231292 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.231265    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg" event={"ID":"26c6dc33-1654-44e2-aa4e-3425a6003376","Type":"ContainerStarted","Data":"279f6d185f0101fcb14c9afa3275acf81647a7c0d455c83e87018d1bac33f115"}
Jul 27 21:05:10.232921 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.232873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss" event={"ID":"138fc1a3-ebe3-45bc-a700-5f6f85a88410","Type":"ContainerStarted","Data":"30dddc6231717f2822f73ed339e37535f5f2658e0cd00b9d4bd2d0e5390f892c"}
Jul 27 21:05:10.235088 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.235063    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016" exitCode=0
Jul 27 21:05:10.235169 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.235141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"}
Jul 27 21:05:10.235221 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.235191    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"684edf42be841522b3e8303600c2635ae5a731ef33d444a6b6bb2367fe9854b1"}
Jul 27 21:05:10.239081 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.239040    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wffr5" podStartSLOduration=67.0074446 podStartE2EDuration="1m13.23902142s" totalImagesPullingTime="6.23157682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:10.23791595 +0000 UTC m=+73.908407601" watchObservedRunningTime="2026-07-27 21:05:10.23902142 +0000 UTC m=+73.909513068"
Jul 27 21:05:10.239450 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.239387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"e776e1abcd37c588bd92ca9f1d6c23b96b21aa5cb84d193ad7b35c199cc46e73"}
Jul 27 21:05:10.239507 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.239449    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"1bbbd8562b6466fa7004b82f5c4a862c17100a13fc0493d9584f7bd42f67c83c"}
Jul 27 21:05:10.239507 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.239490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"9348198cb36db09c5b3fe93daf3026aad698e4128c70db68d97a3ba26759980c"}
Jul 27 21:05:10.291112 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.291043    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d9xcx" podStartSLOduration=68.059069333 podStartE2EDuration="1m14.291024252s" totalImagesPullingTime="6.231954919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:10.290607498 +0000 UTC m=+73.961099148" watchObservedRunningTime="2026-07-27 21:05:10.291024252 +0000 UTC m=+73.961515899"
Jul 27 21:05:10.291706 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.291666    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-77tmg" podStartSLOduration=4.368906932 podStartE2EDuration="9.291658372s" totalImagesPullingTime="4.92275144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:10.264206269 +0000 UTC m=+73.934697918" watchObservedRunningTime="2026-07-27 21:05:10.291658372 +0000 UTC m=+73.962150017"
Jul 27 21:05:10.344970 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:10.344909    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fsfdn" podStartSLOduration=3.920871589 podStartE2EDuration="9.344888767s" totalImagesPullingTime="5.424017178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:10.343618155 +0000 UTC m=+74.014109803" watchObservedRunningTime="2026-07-27 21:05:10.344888767 +0000 UTC m=+74.015380416"
Jul 27 21:05:12.255953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.254949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"}
Jul 27 21:05:12.255953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.255043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"}
Jul 27 21:05:12.255953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.255083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"}
Jul 27 21:05:12.255953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.255113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"}
Jul 27 21:05:12.255953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.255135    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"}
Jul 27 21:05:12.257887 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.257856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss" event={"ID":"138fc1a3-ebe3-45bc-a700-5f6f85a88410","Type":"ContainerStarted","Data":"8d31171af51cdc3251193172c85b07082e5b12773ba0cd2bd205f148766f5f0c"}
Jul 27 21:05:12.258131 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.258108    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:12.259936 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.259832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796887b949-2jbz9" event={"ID":"b20c43c9-df65-4f26-b8b1-749689361936","Type":"ContainerStarted","Data":"45be805fc5b1aa8db75acf5c8fc84b33b87467409db6942ab636f250213a0aef"}
Jul 27 21:05:12.264295 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.264271    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss"
Jul 27 21:05:12.265471 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.265446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"b8c134e835f80119e410d517c165280d71989b650824532f7ad9c7b76915b0ee"}
Jul 27 21:05:12.265567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.265517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"09a712e480b95445333f4ace78983b0a36aa6618ad1cf6ae454df2c574d3cbf7"}
Jul 27 21:05:12.265567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.265561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" event={"ID":"7e60d566-2700-488e-aedd-56107c1a1c4a","Type":"ContainerStarted","Data":"80376d421e57458db194571c91e96bb4d9f09f543bd72ad79385650c022ecd0f"}
Jul 27 21:05:12.265651 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.265605    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:12.292657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.292598    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.229663755 podStartE2EDuration="10.292582706s" totalImagesPullingTime="6.042881342s" totalInitContainerRuntime="20.037609ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:12.289929488 +0000 UTC m=+75.960421135" watchObservedRunningTime="2026-07-27 21:05:12.292582706 +0000 UTC m=+75.963074372"
Jul 27 21:05:12.311580 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.311454    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-796887b949-2jbz9" podStartSLOduration=4.029690044 podStartE2EDuration="6.311432558s" totalImagesPullingTime="2.281742514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:12.310250767 +0000 UTC m=+75.980742415" watchObservedRunningTime="2026-07-27 21:05:12.311432558 +0000 UTC m=+75.981924206"
Jul 27 21:05:12.360186 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.359961    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92" podStartSLOduration=2.862856517 podStartE2EDuration="9.359939303s" totalImagesPullingTime="6.497082786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:12.337512961 +0000 UTC m=+76.008004608" watchObservedRunningTime="2026-07-27 21:05:12.359939303 +0000 UTC m=+76.030430951"
Jul 27 21:05:12.360350 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:12.360179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6mlss" podStartSLOduration=4.600574507 podStartE2EDuration="6.360166842s" totalImagesPullingTime="1.759592335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:12.358437623 +0000 UTC m=+76.028929271" watchObservedRunningTime="2026-07-27 21:05:12.360166842 +0000 UTC m=+76.030658490"
Jul 27 21:05:13.284425 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:13.284379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerStarted","Data":"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"}
Jul 27 21:05:14.295061 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:14.295021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"}
Jul 27 21:05:14.295434 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:14.295093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"}
Jul 27 21:05:14.295434 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:14.295120    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"}
Jul 27 21:05:14.295434 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:14.295146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"}
Jul 27 21:05:15.302764 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:15.302713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"}
Jul 27 21:05:15.303156 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:15.302810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerStarted","Data":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"}
Jul 27 21:05:15.334990 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:15.334931    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.595354287 podStartE2EDuration="7.334908431s" totalImagesPullingTime="3.713343149s" totalInitContainerRuntime="26.210995ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:15.333378243 +0000 UTC m=+79.003869890" watchObservedRunningTime="2026-07-27 21:05:15.334908431 +0000 UTC m=+79.005400074"
Jul 27 21:05:18.290149 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:18.290024    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f94c686b7-9fk92"
Jul 27 21:05:18.412147 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:18.412082    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:05:26.437581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:26.437538    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:26.438116 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:26.437620    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:41.248504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:41.248469    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d9xcx"
Jul 27 21:05:46.443608 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:46.443582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:46.447337 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:46.447315    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-796887b949-2jbz9"
Jul 27 21:05:46.754891 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:05:46.754798    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g58pk_e842bd78-9c4e-47fe-b057-a3ab1db8d36b/serve-healthcheck-canary/0.log"
Jul 27 21:06:08.411572 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:08.411527    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:08.431106 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:08.431081    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:08.533967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:08.533942    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:22.226770 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.226721    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227007    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="alertmanager" containerID="cri-o://6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891" gracePeriod=120
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227072    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-metric" containerID="cri-o://3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc" gracePeriod=120
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227081    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-web" containerID="cri-o://8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170" gracePeriod=120
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227160    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy" containerID="cri-o://7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284" gracePeriod=120
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227170    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="config-reloader" containerID="cri-o://f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782" gracePeriod=120
Jul 27 21:06:22.227296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.227144    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="prom-label-proxy" containerID="cri-o://c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc" gracePeriod=120
Jul 27 21:06:22.580651 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580616    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc" exitCode=0
Jul 27 21:06:22.580651 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580641    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284" exitCode=0
Jul 27 21:06:22.580651 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580647    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782" exitCode=0
Jul 27 21:06:22.580651 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580652    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891" exitCode=0
Jul 27 21:06:22.580934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580687    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"}
Jul 27 21:06:22.580934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580777    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"}
Jul 27 21:06:22.580934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"}
Jul 27 21:06:22.580934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:22.580834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"}
Jul 27 21:06:23.480789 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.480765    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:23.573498 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573474    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573527    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573546    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573569    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573587    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573657 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573639    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573663    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573694    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573770    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573808    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfl9t\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4a309ec8-3d07-4715-bf39-edb8bb31f494\" (UID: \"4a309ec8-3d07-4715-bf39-edb8bb31f494\") "
Jul 27 21:06:23.573946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.573921    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:23.574338 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.574193    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-metrics-client-ca\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.574338 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.574270    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:23.574851 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.574527    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:06:23.576803 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.576771    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.577241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.577200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.577446 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.577419    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out" (OuterVolumeSpecName: "config-out") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:06:23.577673 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.577654    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:23.577800 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.577774    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.577889 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.577862    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume" (OuterVolumeSpecName: "config-volume") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.578025 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.578005    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.578129 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.578114    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t" (OuterVolumeSpecName: "kube-api-access-tfl9t") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "kube-api-access-tfl9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:23.582342 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.582316    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.588944 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.588917    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc" exitCode=0
Jul 27 21:06:23.588944 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.588940    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerID="8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170" exitCode=0
Jul 27 21:06:23.589123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.588991    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:23.589123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.589001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"}
Jul 27 21:06:23.589123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.589071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"}
Jul 27 21:06:23.589123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.589102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a309ec8-3d07-4715-bf39-edb8bb31f494","Type":"ContainerDied","Data":"fe313f0ae15dbb0cb76d75360ec8277b2a4bb8d2b5bb2727bf103a6356599f8e"}
Jul 27 21:06:23.589317 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.589136    2555 scope.go:117] "RemoveContainer" containerID="c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"
Jul 27 21:06:23.589581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.589558    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config" (OuterVolumeSpecName: "web-config") pod "4a309ec8-3d07-4715-bf39-edb8bb31f494" (UID: "4a309ec8-3d07-4715-bf39-edb8bb31f494"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:23.596474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.596454    2555 scope.go:117] "RemoveContainer" containerID="3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"
Jul 27 21:06:23.604492 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.604475    2555 scope.go:117] "RemoveContainer" containerID="7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"
Jul 27 21:06:23.610875 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.610859    2555 scope.go:117] "RemoveContainer" containerID="8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"
Jul 27 21:06:23.617435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.617420    2555 scope.go:117] "RemoveContainer" containerID="f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"
Jul 27 21:06:23.624028 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.624005    2555 scope.go:117] "RemoveContainer" containerID="6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"
Jul 27 21:06:23.630674 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.630658    2555 scope.go:117] "RemoveContainer" containerID="d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"
Jul 27 21:06:23.637330 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.637311    2555 scope.go:117] "RemoveContainer" containerID="c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"
Jul 27 21:06:23.637611 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.637586    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc\": container with ID starting with c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc not found: ID does not exist" containerID="c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"
Jul 27 21:06:23.637663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.637622    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"} err="failed to get container status \"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc\": rpc error: code = NotFound desc = could not find container \"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc\": container with ID starting with c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc not found: ID does not exist"
Jul 27 21:06:23.637663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.637655    2555 scope.go:117] "RemoveContainer" containerID="3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"
Jul 27 21:06:23.637915 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.637898    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc\": container with ID starting with 3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc not found: ID does not exist" containerID="3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"
Jul 27 21:06:23.637984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.637924    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"} err="failed to get container status \"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc\": rpc error: code = NotFound desc = could not find container \"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc\": container with ID starting with 3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc not found: ID does not exist"
Jul 27 21:06:23.637984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.637947    2555 scope.go:117] "RemoveContainer" containerID="7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"
Jul 27 21:06:23.638166 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.638150    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284\": container with ID starting with 7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284 not found: ID does not exist" containerID="7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"
Jul 27 21:06:23.638222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638172    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"} err="failed to get container status \"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284\": rpc error: code = NotFound desc = could not find container \"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284\": container with ID starting with 7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284 not found: ID does not exist"
Jul 27 21:06:23.638222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638192    2555 scope.go:117] "RemoveContainer" containerID="8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"
Jul 27 21:06:23.638388 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.638369    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170\": container with ID starting with 8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170 not found: ID does not exist" containerID="8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"
Jul 27 21:06:23.638429 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638393    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"} err="failed to get container status \"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170\": rpc error: code = NotFound desc = could not find container \"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170\": container with ID starting with 8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170 not found: ID does not exist"
Jul 27 21:06:23.638429 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638416    2555 scope.go:117] "RemoveContainer" containerID="f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"
Jul 27 21:06:23.638645 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.638627    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782\": container with ID starting with f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782 not found: ID does not exist" containerID="f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"
Jul 27 21:06:23.638685 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638650    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"} err="failed to get container status \"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782\": rpc error: code = NotFound desc = could not find container \"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782\": container with ID starting with f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782 not found: ID does not exist"
Jul 27 21:06:23.638685 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638665    2555 scope.go:117] "RemoveContainer" containerID="6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"
Jul 27 21:06:23.638936 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.638917    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891\": container with ID starting with 6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891 not found: ID does not exist" containerID="6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"
Jul 27 21:06:23.638990 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638940    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"} err="failed to get container status \"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891\": rpc error: code = NotFound desc = could not find container \"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891\": container with ID starting with 6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891 not found: ID does not exist"
Jul 27 21:06:23.638990 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.638954    2555 scope.go:117] "RemoveContainer" containerID="d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"
Jul 27 21:06:23.639198 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:23.639182    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60\": container with ID starting with d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60 not found: ID does not exist" containerID="d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"
Jul 27 21:06:23.639243 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639202    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"} err="failed to get container status \"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60\": rpc error: code = NotFound desc = could not find container \"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60\": container with ID starting with d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60 not found: ID does not exist"
Jul 27 21:06:23.639243 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639215    2555 scope.go:117] "RemoveContainer" containerID="c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"
Jul 27 21:06:23.639445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639428    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc"} err="failed to get container status \"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc\": rpc error: code = NotFound desc = could not find container \"c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc\": container with ID starting with c7943d45c9cc0f95d38f380b4d78411f05b1bd4c26f5772f42dbb99de2cd0fdc not found: ID does not exist"
Jul 27 21:06:23.639445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639445    2555 scope.go:117] "RemoveContainer" containerID="3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"
Jul 27 21:06:23.639655 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639640    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc"} err="failed to get container status \"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc\": rpc error: code = NotFound desc = could not find container \"3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc\": container with ID starting with 3deb47d5b702267d11ee8ddfbfc2052b1422a2599f581a00eee03b6448d755bc not found: ID does not exist"
Jul 27 21:06:23.639696 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639656    2555 scope.go:117] "RemoveContainer" containerID="7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"
Jul 27 21:06:23.639854 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639839    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284"} err="failed to get container status \"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284\": rpc error: code = NotFound desc = could not find container \"7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284\": container with ID starting with 7fb862706727f242b604faa2ecd170df0b3bd6fac870922dced024aa3c617284 not found: ID does not exist"
Jul 27 21:06:23.639964 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.639854    2555 scope.go:117] "RemoveContainer" containerID="8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"
Jul 27 21:06:23.640033 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640016    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170"} err="failed to get container status \"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170\": rpc error: code = NotFound desc = could not find container \"8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170\": container with ID starting with 8bc9974836f3213a06d9a763ee1af01ce06eb4b7226db1bc12fc9f0303d04170 not found: ID does not exist"
Jul 27 21:06:23.640072 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640034    2555 scope.go:117] "RemoveContainer" containerID="f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"
Jul 27 21:06:23.640244 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640227    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782"} err="failed to get container status \"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782\": rpc error: code = NotFound desc = could not find container \"f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782\": container with ID starting with f7fb2410742a7b2a3cce552165613c87ff306c48f6048644a2d66cda1cd78782 not found: ID does not exist"
Jul 27 21:06:23.640244 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640244    2555 scope.go:117] "RemoveContainer" containerID="6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"
Jul 27 21:06:23.640421 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640405    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891"} err="failed to get container status \"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891\": rpc error: code = NotFound desc = could not find container \"6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891\": container with ID starting with 6e22e8fb7bdc985b0386ce753e9c41584420959fd998d9c0aac81aab8fd93891 not found: ID does not exist"
Jul 27 21:06:23.640459 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640423    2555 scope.go:117] "RemoveContainer" containerID="d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"
Jul 27 21:06:23.640613 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.640597    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60"} err="failed to get container status \"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60\": rpc error: code = NotFound desc = could not find container \"d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60\": container with ID starting with d13eb8b2e6eb92cb77bed7ca0c54c3a0a16a4d62b11ea066ff89cdd1dc6acd60 not found: ID does not exist"
Jul 27 21:06:23.675297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675265    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfl9t\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-kube-api-access-tfl9t\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675291    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675305    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a309ec8-3d07-4715-bf39-edb8bb31f494-tls-assets\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675314    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675324    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-main-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675333    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-out\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675341    2555 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-config-volume\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675351    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-web-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675359    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675367    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675376    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a309ec8-3d07-4715-bf39-edb8bb31f494-alertmanager-main-db\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.675501 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.675384    2555 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a309ec8-3d07-4715-bf39-edb8bb31f494-cluster-tls-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:23.917850 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.917819    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:23.924107 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.924046    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:23.960935 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.960903    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:23.961242 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961228    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="init-config-reloader"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961244    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="init-config-reloader"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961257    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961263    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961270    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-metric"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-metric"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961282    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="prom-label-proxy"
Jul 27 21:06:23.961297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961290    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="prom-label-proxy"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961303    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-web"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961308    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-web"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961316    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="alertmanager"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961321    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="alertmanager"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961334    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="config-reloader"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961341    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="config-reloader"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961389    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="prom-label-proxy"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961400    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="config-reloader"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961406    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-web"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961412    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="alertmanager"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy-metric"
Jul 27 21:06:23.961503 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.961424    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" containerName="kube-rbac-proxy"
Jul 27 21:06:23.967610 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.967577    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:23.970879 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.970850    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mldfb\""
Jul 27 21:06:23.970995 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.970856    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:06:23.971915 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.971897    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:06:23.972512 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.972486    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:06:23.972614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.972492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:06:23.972897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.972880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:06:23.973000 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.972885    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:06:23.973000 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.972938    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:06:23.973970 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.973953    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:06:23.980255 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.980073    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:06:23.982120 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:23.982099    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:24.078773 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-web-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.078773 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.078991 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rf2\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-kube-api-access-d7rf2\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.078991 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.078991 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078888    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-config-volume\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.078991 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079124 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.078994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079124 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-config-out\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079124 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079044    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079124 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.079299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.079160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.179925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-config-out\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.179925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.179925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179941    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.179993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-web-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rf2\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-kube-api-access-d7rf2\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-config-volume\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180219 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180650 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180229    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.180650 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.180429    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.181631 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.181595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.182227 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.182198    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1875ec7e-3d46-48b8-b007-a37707263403-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.182946 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.182921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1875ec7e-3d46-48b8-b007-a37707263403-config-out\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183045 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183188 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183333 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183397 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-config-volume\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183667 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183644    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.183732 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.183719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.184707 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.184684    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.184810 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.184717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1875ec7e-3d46-48b8-b007-a37707263403-web-config\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.192169 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.192130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rf2\" (UniqueName: \"kubernetes.io/projected/1875ec7e-3d46-48b8-b007-a37707263403-kube-api-access-d7rf2\") pod \"alertmanager-main-0\" (UID: \"1875ec7e-3d46-48b8-b007-a37707263403\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.278541 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.278501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:24.458292 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.458207    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:24.462156 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:06:24.462130    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1875ec7e_3d46_48b8_b007_a37707263403.slice/crio-712247ff38489fb44e4f929b8b2f77dcd4c7c0bf73aa939542f34e37ba06d072 WatchSource:0}: Error finding container 712247ff38489fb44e4f929b8b2f77dcd4c7c0bf73aa939542f34e37ba06d072: Status 404 returned error can't find the container with id 712247ff38489fb44e4f929b8b2f77dcd4c7c0bf73aa939542f34e37ba06d072
Jul 27 21:06:24.595533 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.595501    2555 generic.go:356] "Generic (PLEG): container finished" podID="1875ec7e-3d46-48b8-b007-a37707263403" containerID="c114849158109db5544cdff1794e3d82a89b3023cf9f339e7bf9986a492eda74" exitCode=0
Jul 27 21:06:24.595937 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.595591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerDied","Data":"c114849158109db5544cdff1794e3d82a89b3023cf9f339e7bf9986a492eda74"}
Jul 27 21:06:24.595937 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.595663    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"712247ff38489fb44e4f929b8b2f77dcd4c7c0bf73aa939542f34e37ba06d072"}
Jul 27 21:06:24.858172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:24.858139    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a309ec8-3d07-4715-bf39-edb8bb31f494" path="/var/lib/kubelet/pods/4a309ec8-3d07-4715-bf39-edb8bb31f494/volumes"
Jul 27 21:06:25.604560 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"1544740e3dc56635e37a825841cb20ebabce83245cfe78acf23d979e0a2bedc3"}
Jul 27 21:06:25.604953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"de6e94ecc12384fd703067e95ad6e51b723233b3d188dd46871ee5316ec34a8a"}
Jul 27 21:06:25.604953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"159a2f106898aa4b95b7471329d976596829870a5733e39d06cbaa4c2d6f979d"}
Jul 27 21:06:25.604953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"280a62d17a41e1b2c06502e22c2dad831cf8d3ff131726febb265a90fd4da2b1"}
Jul 27 21:06:25.604953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"d2129866f9398eb785f1af83754c2e8f945f228b603e70cd2358efd28ef5b000"}
Jul 27 21:06:25.604953 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.604775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1875ec7e-3d46-48b8-b007-a37707263403","Type":"ContainerStarted","Data":"83d2153c436c8d99bbf99d60603721ec8bb9e57e00f834c437352f9f7c361153"}
Jul 27 21:06:25.654838 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:25.654784    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.63863934 podStartE2EDuration="2.654769191s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.129851ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:25.653857087 +0000 UTC m=+149.324348734" watchObservedRunningTime="2026-07-27 21:06:25.654769191 +0000 UTC m=+149.325260829"
Jul 27 21:06:26.760928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.760891    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:26.761340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761228    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="prometheus" containerID="cri-o://fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10" gracePeriod=600
Jul 27 21:06:26.761340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761251    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy" containerID="cri-o://945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd" gracePeriod=600
Jul 27 21:06:26.761340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761287    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="config-reloader" containerID="cri-o://3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff" gracePeriod=600
Jul 27 21:06:26.761340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761308    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-web" containerID="cri-o://e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f" gracePeriod=600
Jul 27 21:06:26.761536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761350    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674" gracePeriod=600
Jul 27 21:06:26.761638 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:26.761283    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="thanos-sidecar" containerID="cri-o://0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f" gracePeriod=600
Jul 27 21:06:27.006482 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.005447    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.104160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104130    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104166    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104186    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104206    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vgjw\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104352    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104396    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104434    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104459    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104489    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104513    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104558    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104565    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104582    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104695    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104723    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.104820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104775    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle\") pod \"66ad960c-2763-437a-992f-7b7ad9c9aea1\" (UID: \"66ad960c-2763-437a-992f-7b7ad9c9aea1\") "
Jul 27 21:06:27.105322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.104880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.105322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.105136    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.105322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.105157    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.105322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.105168    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.105939 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.105650    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:06:27.106943 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.106915    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.107422 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.107118    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.107495 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.107418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.107976 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.107698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.108897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.108674    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out" (OuterVolumeSpecName: "config-out") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:06:27.109138 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109065    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.109138 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109094    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:27.109397 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109253    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.109397 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109282    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:27.109529 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109466    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.109581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109535    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.110080 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.109992    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.110169 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.110128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.110579 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.110559    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw" (OuterVolumeSpecName: "kube-api-access-6vgjw") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "kube-api-access-6vgjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:27.111103 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.111078    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config" (OuterVolumeSpecName: "config") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.119388 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.119361    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config" (OuterVolumeSpecName: "web-config") pod "66ad960c-2763-437a-992f-7b7ad9c9aea1" (UID: "66ad960c-2763-437a-992f-7b7ad9c9aea1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:27.205831 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205786    2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-kube-rbac-proxy\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.205831 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205823    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-db\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.205831 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205836    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgjw\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-kube-api-access-6vgjw\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205849    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205862    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-metrics-client-ca\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205874    2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-metrics-client-certs\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205886    2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205896    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ad960c-2763-437a-992f-7b7ad9c9aea1-tls-assets\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205907    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205919    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205931    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205942    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-web-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205952    2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205963    2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-grpc-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205975    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205986    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ad960c-2763-437a-992f-7b7ad9c9aea1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.205998    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ad960c-2763-437a-992f-7b7ad9c9aea1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.206105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.206009    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ad960c-2763-437a-992f-7b7ad9c9aea1-config-out\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:06:27.620825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620793    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674" exitCode=0
Jul 27 21:06:27.620825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620818    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd" exitCode=0
Jul 27 21:06:27.620825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620825    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f" exitCode=0
Jul 27 21:06:27.620825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620830    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f" exitCode=0
Jul 27 21:06:27.620825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620835    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff" exitCode=0
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620841    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10" exitCode=0
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"}
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620880    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.620982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"}
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"}
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621046    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.621117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"}
Jul 27 21:06:27.621369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"}
Jul 27 21:06:27.621369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621164    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"}
Jul 27 21:06:27.621369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.621203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ad960c-2763-437a-992f-7b7ad9c9aea1","Type":"ContainerDied","Data":"684edf42be841522b3e8303600c2635ae5a731ef33d444a6b6bb2367fe9854b1"}
Jul 27 21:06:27.628930 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.628905    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.635820 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.635802    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.643356 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.643336    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.650502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.650481    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.653506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.653484    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:27.657857 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.657840    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.665571 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.665548    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.666499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.666431    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:27.674679 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.674659    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.675237 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.675200    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.675297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675245    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.675297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675265    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.675536 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.675516    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.675599 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675545    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.675599 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675564    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.675859 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.675836    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.675952 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675868    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.675952 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.675897    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.676132 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.676115    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.676191 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676140    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.676191 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676163    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.676410 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.676393    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.676449 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676415    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.676449 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676430    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.676662 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.676645    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.676721 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676669    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.676721 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676688    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.676933 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:06:27.676918    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.676999 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676940    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.676999 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.676958    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.677190 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677172    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.677241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677192    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.677409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677392    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.677455 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677410    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.677632 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677615    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.677710 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677634    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.677900 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677881    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.677967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.677902    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.678109 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678091    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.678204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678110    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.678349 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678334    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.678349 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678349    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.678540 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678525    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.678577 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678540    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.678728 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678714    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.678794 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678728    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.678947 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678932    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.678982 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.678947    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.679125 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679110    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.679181 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679125    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.679357 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679339    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.679406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679358    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.679533 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679514    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.679630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679549    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.679788 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679766    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.679876 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.679788    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.680036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680017    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.680084 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680037    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.680253 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680237    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.680253 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680253    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.680464 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680449    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.680516 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680464    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.680663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680646    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.680706 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680663    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.680876 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680861    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.680924 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.680876    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.681054 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681038    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.681098 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681054    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.681226 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681210    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.681271 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681226    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.681385 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681370    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.681431 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681386    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.681581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681565    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.681630 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681581    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.681762 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681733    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.681805 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681766    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.681951 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681935    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.681989 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.681952    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.682120 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682104    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.682161 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682120    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.682340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682318    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.682340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682341    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.682534 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682519    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.682578 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682534    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.682718 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682703    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.682780 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682719    2555 scope.go:117] "RemoveContainer" containerID="6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"
Jul 27 21:06:27.682948 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682917    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674"} err="failed to get container status \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": rpc error: code = NotFound desc = could not find container \"6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674\": container with ID starting with 6bc44d78049f35e34ef02335b6c1c7e094c78e44264f2cf450de234138889674 not found: ID does not exist"
Jul 27 21:06:27.683023 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.682950    2555 scope.go:117] "RemoveContainer" containerID="945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"
Jul 27 21:06:27.683177 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683159    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd"} err="failed to get container status \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": rpc error: code = NotFound desc = could not find container \"945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd\": container with ID starting with 945c492f64088b6898ac8214d6804337bb5a5517a4106485e46ebac68b92e6fd not found: ID does not exist"
Jul 27 21:06:27.683226 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683178    2555 scope.go:117] "RemoveContainer" containerID="e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"
Jul 27 21:06:27.683385 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683366    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f"} err="failed to get container status \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": rpc error: code = NotFound desc = could not find container \"e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f\": container with ID starting with e9a04bdf47bd2c265f56ce8ebac296a394be067fae4eec041a1b82240774d19f not found: ID does not exist"
Jul 27 21:06:27.683445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683386    2555 scope.go:117] "RemoveContainer" containerID="0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"
Jul 27 21:06:27.683596 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683579    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f"} err="failed to get container status \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": rpc error: code = NotFound desc = could not find container \"0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f\": container with ID starting with 0f5342eef6b3757691e56a00112f73700e9068d8eb7f48e9f0b8c2521190ba0f not found: ID does not exist"
Jul 27 21:06:27.683636 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683597    2555 scope.go:117] "RemoveContainer" containerID="3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"
Jul 27 21:06:27.683823 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683806    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff"} err="failed to get container status \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": rpc error: code = NotFound desc = could not find container \"3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff\": container with ID starting with 3cae9c259fc752621eb9211193a325f73db700817d2250a437fced320e921eff not found: ID does not exist"
Jul 27 21:06:27.683875 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.683824    2555 scope.go:117] "RemoveContainer" containerID="fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"
Jul 27 21:06:27.684035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.684019    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10"} err="failed to get container status \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": rpc error: code = NotFound desc = could not find container \"fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10\": container with ID starting with fb577cda9f0d8164338b72ba1ce1ea48fb0922f7fd033c63b18892d8d6629b10 not found: ID does not exist"
Jul 27 21:06:27.684099 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.684036    2555 scope.go:117] "RemoveContainer" containerID="edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"
Jul 27 21:06:27.684281 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.684260    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016"} err="failed to get container status \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": rpc error: code = NotFound desc = could not find container \"edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016\": container with ID starting with edb39824f4b24aec3a97aaf1eac01f6e011ad50c35ce6dc0917bfb18af2b7016 not found: ID does not exist"
Jul 27 21:06:27.727217 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727185    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:27.727492 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727480    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-web"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727494    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-web"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727503    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="thanos-sidecar"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727508    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="thanos-sidecar"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727515    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-thanos"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727521    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-thanos"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727534    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727540    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727550    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="init-config-reloader"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727558    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="init-config-reloader"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727567    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="config-reloader"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727574    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="config-reloader"
Jul 27 21:06:27.727582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727587    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="prometheus"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727592    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="prometheus"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727638    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-web"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727645    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="prometheus"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727650    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy-thanos"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727658    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="kube-rbac-proxy"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727664    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="thanos-sidecar"
Jul 27 21:06:27.728030 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.727670    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" containerName="config-reloader"
Jul 27 21:06:27.731458 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.731437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.744803 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.744780    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:06:27.744965 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.744947    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:06:27.745079 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:06:27.745079 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745058    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:06:27.745079 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745066    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:06:27.745273 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745254    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:06:27.745374 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:06:27.745374 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745331    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:06:27.745516 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745393    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:06:27.745612 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745596    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:06:27.745660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745596    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:06:27.745660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-47i89gm7bch3\""
Jul 27 21:06:27.745727 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.745628    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xqzbz\""
Jul 27 21:06:27.746362 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.746321    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:06:27.750048 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.750025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:06:27.756180 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.756159    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:06:27.768454 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.768422    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:27.811836 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.811803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.811836 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.811842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.811864    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.811915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.811976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-web-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqcb\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-kube-api-access-swqcb\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812056 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812075    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-config-out\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812264 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812299    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.812473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.812411    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913006    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-config-out\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913353 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913353 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913353 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913385    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913622 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913598    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913741 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913741 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913969 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913969 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913969 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913933    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-web-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.913969 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqcb\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-kube-api-access-swqcb\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.914150 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.914150 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.914027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.914150 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.914054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.914150 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.914112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.914150 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.914142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.914971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.915021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.915967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-config-out\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.916281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.916293    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.916305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.913605    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916852 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.916498    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.916852 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.916780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.917297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.917271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.917602 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.917579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.917689 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.917675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.918159 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.918139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.918925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.918907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-web-config\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.919007 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.918962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.919252 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.919231    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e258328b-05ca-479a-bb31-35f38b079741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.920032 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.920015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e258328b-05ca-479a-bb31-35f38b079741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:27.925028 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:27.925010    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqcb\" (UniqueName: \"kubernetes.io/projected/e258328b-05ca-479a-bb31-35f38b079741-kube-api-access-swqcb\") pod \"prometheus-k8s-0\" (UID: \"e258328b-05ca-479a-bb31-35f38b079741\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:28.040551 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.040516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:28.192875 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.192792    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:28.195541 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:06:28.195510    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode258328b_05ca_479a_bb31_35f38b079741.slice/crio-53128fd42513c2e5cf3fc97866a6c4946e2b6b796f6f524c6de6a75b49c12aab WatchSource:0}: Error finding container 53128fd42513c2e5cf3fc97866a6c4946e2b6b796f6f524c6de6a75b49c12aab: Status 404 returned error can't find the container with id 53128fd42513c2e5cf3fc97866a6c4946e2b6b796f6f524c6de6a75b49c12aab
Jul 27 21:06:28.626125 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.626087    2555 generic.go:356] "Generic (PLEG): container finished" podID="e258328b-05ca-479a-bb31-35f38b079741" containerID="a91b727a31562003904f1997f21d8d72eb4ee3508f9f3bc92c450f6e42c8c8fe" exitCode=0
Jul 27 21:06:28.626299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.626178    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerDied","Data":"a91b727a31562003904f1997f21d8d72eb4ee3508f9f3bc92c450f6e42c8c8fe"}
Jul 27 21:06:28.626299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.626249    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"53128fd42513c2e5cf3fc97866a6c4946e2b6b796f6f524c6de6a75b49c12aab"}
Jul 27 21:06:28.863643 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:28.863603    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ad960c-2763-437a-992f-7b7ad9c9aea1" path="/var/lib/kubelet/pods/66ad960c-2763-437a-992f-7b7ad9c9aea1/volumes"
Jul 27 21:06:29.634559 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"540dea24011a475e438fc81382d98f81d01478f0b447ad8b774c65d9c1f55796"}
Jul 27 21:06:29.634709 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"b416f115591c6f806e5d68bb10e0d74dcc5c76bed21da032934556b5e2025c82"}
Jul 27 21:06:29.634709 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634613    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"8b14eda233cfe4dee8900731c3c97894f287a1c51e455a20a58ad4ab6a7c156e"}
Jul 27 21:06:29.634709 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634637    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"c824ab6ad966c59c92774eb8995a9ef59406d53788034053ffdc219ef7c8695e"}
Jul 27 21:06:29.634709 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"2eb72e9236915d118dde47c3b28baf080a239a954cddc97e317e1fbe8b27c01c"}
Jul 27 21:06:29.634709 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.634683    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e258328b-05ca-479a-bb31-35f38b079741","Type":"ContainerStarted","Data":"9f25401967781a9374ca2e2e72e35f6b5c93d898c8d97bbc8a4554d7a31493af"}
Jul 27 21:06:29.672692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:29.672635    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6537946249999997 podStartE2EDuration="2.672617049s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.822424ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:29.670003623 +0000 UTC m=+153.340495309" watchObservedRunningTime="2026-07-27 21:06:29.672617049 +0000 UTC m=+153.343108696"
Jul 27 21:06:33.041624 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:06:33.041586    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:28.041807 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:07:28.041742    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:28.057178 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:07:28.057147    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:28.888845 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:07:28.888817    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:08:56.743287 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:08:56.743256    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:11:26.505118 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.505038    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-7ghbb"]
Jul 27 21:11:26.508432 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.508412    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.513068 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.513048    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 21:11:26.513642 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.513620    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:11:26.514856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.514550    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:11:26.514856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.514792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tj2ts\""
Jul 27 21:11:26.522367 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.522346    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7ghbb"]
Jul 27 21:11:26.587411 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.587378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-data\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.587411 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.587412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lm2\" (UniqueName: \"kubernetes.io/projected/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-kube-api-access-c2lm2\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.688582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.688549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-data\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.688582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.688583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lm2\" (UniqueName: \"kubernetes.io/projected/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-kube-api-access-c2lm2\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.688947 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.688928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-data\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.702875 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.702847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lm2\" (UniqueName: \"kubernetes.io/projected/f5cfa3e6-cf81-48c8-ab66-faa8fe982425-kube-api-access-c2lm2\") pod \"seaweedfs-64568bcd49-7ghbb\" (UID: \"f5cfa3e6-cf81-48c8-ab66-faa8fe982425\") " pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.817010 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.816927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:26.976432 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.976397    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7ghbb"]
Jul 27 21:11:26.979447 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:11:26.979417    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cfa3e6_cf81_48c8_ab66_faa8fe982425.slice/crio-1aba6eb900dce600349222aeeea6f4668040874b83177d14ebc01b5d29415d81 WatchSource:0}: Error finding container 1aba6eb900dce600349222aeeea6f4668040874b83177d14ebc01b5d29415d81: Status 404 returned error can't find the container with id 1aba6eb900dce600349222aeeea6f4668040874b83177d14ebc01b5d29415d81
Jul 27 21:11:26.981232 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:26.981212    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:11:27.798691 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:27.798653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7ghbb" event={"ID":"f5cfa3e6-cf81-48c8-ab66-faa8fe982425","Type":"ContainerStarted","Data":"1aba6eb900dce600349222aeeea6f4668040874b83177d14ebc01b5d29415d81"}
Jul 27 21:11:30.812808 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:30.812742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7ghbb" event={"ID":"f5cfa3e6-cf81-48c8-ab66-faa8fe982425","Type":"ContainerStarted","Data":"2eba8b551834b063e9e02d5f72150791871a873a46c50b088fb6ed236c62831e"}
Jul 27 21:11:30.813214 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:30.812864    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:11:30.828670 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:30.828616    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-7ghbb" podStartSLOduration=1.6003718519999999 podStartE2EDuration="4.82860037s" totalImagesPullingTime="3.228228518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:30.828075376 +0000 UTC m=+454.498567024" watchObservedRunningTime="2026-07-27 21:11:30.82860037 +0000 UTC m=+454.499092018"
Jul 27 21:11:36.818664 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:11:36.818627    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-7ghbb"
Jul 27 21:12:44.842409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.842367    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-wpn6v"]
Jul 27 21:12:44.845911 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.845869    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:44.848374 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.848351    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 27 21:12:44.848534 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.848520    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2q59q\""
Jul 27 21:12:44.859223 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.859202    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wpn6v"]
Jul 27 21:12:44.860521 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.860503    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-2pqqc"]
Jul 27 21:12:44.863999 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.863979    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:44.866607 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.866583    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 27 21:12:44.866705 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.866652    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8dl6m\""
Jul 27 21:12:44.875538 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.875517    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-2pqqc"]
Jul 27 21:12:44.968871 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.968833    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:44.969076 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.968890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2745749-5692-43c4-add5-458b9adf7900-tls-certs\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:44.969139 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.969098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmjk\" (UniqueName: \"kubernetes.io/projected/b2745749-5692-43c4-add5-458b9adf7900-kube-api-access-jcmjk\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:44.969209 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:44.969191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v8l\" (UniqueName: \"kubernetes.io/projected/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-kube-api-access-62v8l\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:45.069938 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.069890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:45.069938 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.069951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2745749-5692-43c4-add5-458b9adf7900-tls-certs\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:45.070190 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.069970    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmjk\" (UniqueName: \"kubernetes.io/projected/b2745749-5692-43c4-add5-458b9adf7900-kube-api-access-jcmjk\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:45.070190 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.070008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v8l\" (UniqueName: \"kubernetes.io/projected/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-kube-api-access-62v8l\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:45.070190 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:12:45.070043    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:45.070190 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:12:45.070113    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert podName:826ae9c4-a8d8-4e3e-adb3-28203b7a9b54 nodeName:}" failed. No retries permitted until 2026-07-27 21:12:45.570093265 +0000 UTC m=+529.240584890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert") pod "odh-model-controller-75d9cdbddf-2pqqc" (UID: "826ae9c4-a8d8-4e3e-adb3-28203b7a9b54") : secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:45.072435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.072403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2745749-5692-43c4-add5-458b9adf7900-tls-certs\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:45.079644 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.079622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmjk\" (UniqueName: \"kubernetes.io/projected/b2745749-5692-43c4-add5-458b9adf7900-kube-api-access-jcmjk\") pod \"model-serving-api-7b954ffc95-wpn6v\" (UID: \"b2745749-5692-43c4-add5-458b9adf7900\") " pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:45.079767 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.079709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v8l\" (UniqueName: \"kubernetes.io/projected/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-kube-api-access-62v8l\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:45.157909 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.157819    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:45.300238 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.300188    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wpn6v"]
Jul 27 21:12:45.302822 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:12:45.302793    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2745749_5692_43c4_add5_458b9adf7900.slice/crio-2824f1c203b08c41d7bf60136a139040c71cc968719e7c382b8c6c8acd0b8d56 WatchSource:0}: Error finding container 2824f1c203b08c41d7bf60136a139040c71cc968719e7c382b8c6c8acd0b8d56: Status 404 returned error can't find the container with id 2824f1c203b08c41d7bf60136a139040c71cc968719e7c382b8c6c8acd0b8d56
Jul 27 21:12:45.574988 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:45.574955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:45.575142 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:12:45.575116    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:45.575189 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:12:45.575179    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert podName:826ae9c4-a8d8-4e3e-adb3-28203b7a9b54 nodeName:}" failed. No retries permitted until 2026-07-27 21:12:46.575160429 +0000 UTC m=+530.245652061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert") pod "odh-model-controller-75d9cdbddf-2pqqc" (UID: "826ae9c4-a8d8-4e3e-adb3-28203b7a9b54") : secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:46.117803 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:46.117764    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wpn6v" event={"ID":"b2745749-5692-43c4-add5-458b9adf7900","Type":"ContainerStarted","Data":"2824f1c203b08c41d7bf60136a139040c71cc968719e7c382b8c6c8acd0b8d56"}
Jul 27 21:12:46.585261 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:46.585226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:46.588211 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:46.588180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/826ae9c4-a8d8-4e3e-adb3-28203b7a9b54-cert\") pod \"odh-model-controller-75d9cdbddf-2pqqc\" (UID: \"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54\") " pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:46.674251 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:46.674206    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:46.821764 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:46.821712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-2pqqc"]
Jul 27 21:12:47.743783 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:12:47.743727    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826ae9c4_a8d8_4e3e_adb3_28203b7a9b54.slice/crio-0d0bb94bdea8fade78bba301f669b0a4976940c7f56aa34ad6293328896e52c9 WatchSource:0}: Error finding container 0d0bb94bdea8fade78bba301f669b0a4976940c7f56aa34ad6293328896e52c9: Status 404 returned error can't find the container with id 0d0bb94bdea8fade78bba301f669b0a4976940c7f56aa34ad6293328896e52c9
Jul 27 21:12:48.127874 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:48.127839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wpn6v" event={"ID":"b2745749-5692-43c4-add5-458b9adf7900","Type":"ContainerStarted","Data":"1801b2679e0b6e127950c9e1bf80bc9f637796cc2d348b81cada5557d68d3e2f"}
Jul 27 21:12:48.128084 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:48.127939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:12:48.128859 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:48.128841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc" event={"ID":"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54","Type":"ContainerStarted","Data":"0d0bb94bdea8fade78bba301f669b0a4976940c7f56aa34ad6293328896e52c9"}
Jul 27 21:12:48.147518 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:48.147465    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-wpn6v" podStartSLOduration=1.659050597 podStartE2EDuration="4.147450599s" totalImagesPullingTime="2.488400002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:48.145960751 +0000 UTC m=+531.816452401" watchObservedRunningTime="2026-07-27 21:12:48.147450599 +0000 UTC m=+531.817942247"
Jul 27 21:12:51.143616 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:51.143579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc" event={"ID":"826ae9c4-a8d8-4e3e-adb3-28203b7a9b54","Type":"ContainerStarted","Data":"ae18c8de6242cf3d651e06c9455c8dca103dc4158a85623759edeb41f3b0d860"}
Jul 27 21:12:51.144027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:51.143707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:12:51.161029 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:51.160965    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc" podStartSLOduration=4.50361117 podStartE2EDuration="7.160946162s" totalImagesPullingTime="2.657334992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:51.160638198 +0000 UTC m=+534.831129847" watchObservedRunningTime="2026-07-27 21:12:51.160946162 +0000 UTC m=+534.831437810"
Jul 27 21:12:59.138225 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:12:59.138191    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-wpn6v"
Jul 27 21:13:02.150668 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:02.150638    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-2pqqc"
Jul 27 21:13:03.003598 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.003550    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gmjzz"]
Jul 27 21:13:03.007671 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.007653    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.013868 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.013838    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gmjzz"]
Jul 27 21:13:03.033792 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.033743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.033963 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.033893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4xx\" (UniqueName: \"kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.134392 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.134349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4xx\" (UniqueName: \"kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.134601 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.134409    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.134857 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.134833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.145331 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.145303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4xx\" (UniqueName: \"kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx\") pod \"s3-init-gmjzz\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") " pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.323389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.323360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gmjzz"
Jul 27 21:13:03.452872 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:03.452850    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gmjzz"]
Jul 27 21:13:03.455713 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:13:03.455689    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda60e6a8f_6118_43e6_ba87_80a85b512c1b.slice/crio-d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3 WatchSource:0}: Error finding container d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3: Status 404 returned error can't find the container with id d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3
Jul 27 21:13:04.199973 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:04.199935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerStarted","Data":"d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3"}
Jul 27 21:13:08.220227 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:08.220136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerStarted","Data":"78f64b9f4e3e7bd7aa55c179ee3219d0f9d44db60738ccce57d7b5e80059cf46"}
Jul 27 21:13:10.230443 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:10.230411    2555 generic.go:356] "Generic (PLEG): container finished" podID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerID="78f64b9f4e3e7bd7aa55c179ee3219d0f9d44db60738ccce57d7b5e80059cf46" exitCode=0
Jul 27 21:13:10.230807 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:10.230490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerDied","Data":"78f64b9f4e3e7bd7aa55c179ee3219d0f9d44db60738ccce57d7b5e80059cf46"}
Jul 27 21:13:15.256788 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:15.256743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerStarted","Data":"c83a0639944865be3e6c27b2f2c31594d3c7e5e4145dc2a0fd5ac9b4587c12b4"}
Jul 27 21:13:15.273695 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:13:15.273643    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gmjzz" podStartSLOduration=3.17049131 podStartE2EDuration="13.273628476s" totalImagesPullingTime="8.17501897s" totalInitContainerRuntime="1.928118196s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:13:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:13:15.273293322 +0000 UTC m=+558.943784969" watchObservedRunningTime="2026-07-27 21:13:15.273628476 +0000 UTC m=+558.944120122"
Jul 27 21:14:19.523208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:19.523166    2555 generic.go:356] "Generic (PLEG): container finished" podID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerID="c83a0639944865be3e6c27b2f2c31594d3c7e5e4145dc2a0fd5ac9b4587c12b4" exitCode=0
Jul 27 21:14:19.523608 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:19.523224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerDied","Data":"c83a0639944865be3e6c27b2f2c31594d3c7e5e4145dc2a0fd5ac9b4587c12b4"}
Jul 27 21:14:20.651481 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.651457    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gmjzz"
Jul 27 21:14:20.789700 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.789614    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4xx\" (UniqueName: \"kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx\") pod \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") "
Jul 27 21:14:20.789700 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.789664    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache\") pod \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\" (UID: \"a60e6a8f-6118-43e6-ba87-80a85b512c1b\") "
Jul 27 21:14:20.791870 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.791844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx" (OuterVolumeSpecName: "kube-api-access-rz4xx") pod "a60e6a8f-6118-43e6-ba87-80a85b512c1b" (UID: "a60e6a8f-6118-43e6-ba87-80a85b512c1b"). InnerVolumeSpecName "kube-api-access-rz4xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:14:20.808195 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.808164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a60e6a8f-6118-43e6-ba87-80a85b512c1b" (UID: "a60e6a8f-6118-43e6-ba87-80a85b512c1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:14:20.890364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.890330    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4xx\" (UniqueName: \"kubernetes.io/projected/a60e6a8f-6118-43e6-ba87-80a85b512c1b-kube-api-access-rz4xx\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:14:20.890364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:20.890360    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60e6a8f-6118-43e6-ba87-80a85b512c1b-model-cache\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:14:21.533293 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:21.533261    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gmjzz"
Jul 27 21:14:21.533465 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:21.533261    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gmjzz" event={"ID":"a60e6a8f-6118-43e6-ba87-80a85b512c1b","Type":"ContainerDied","Data":"d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3"}
Jul 27 21:14:21.533465 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:21.533378    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ba00a116a22611a9f5bbcb0ff8619f3b2938544d13c27fc12791aefe5202a3"
Jul 27 21:14:34.334493 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334447    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:14:34.334967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334787    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerName="download-hf-model"
Jul 27 21:14:34.334967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334798    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerName="download-hf-model"
Jul 27 21:14:34.334967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334814    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerName="s3-init"
Jul 27 21:14:34.334967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerName="s3-init"
Jul 27 21:14:34.334967 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.334877    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60e6a8f-6118-43e6-ba87-80a85b512c1b" containerName="s3-init"
Jul 27 21:14:34.338315 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.338298    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.341270 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.341241    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:14:34.341405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.341300    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:14:34.341405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.341335    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 27 21:14:34.341405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.341303    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:14:34.341405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.341373    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 27 21:14:34.342561 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.342541    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\""
Jul 27 21:14:34.342695 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.342563    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:14:34.349830 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.349806    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:14:34.503818 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.503707    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:14:34.509684 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.507938    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.512043 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.512022    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-070ff-predictor-serving-cert\""
Jul 27 21:14:34.512043 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.512037    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-070ff-kube-rbac-proxy-sar-config\""
Jul 27 21:14:34.516959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.516932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.517107 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.517032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.517107 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.517067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.517222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.517133    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvtp\" (UniqueName: \"kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.517222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.517191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.518404 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.518386    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:14:34.618149 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxll\" (UniqueName: \"kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvtp\" (UniqueName: \"kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.618606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618439    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.618988 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.618943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.619221 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.619187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.619415 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.619396    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.621081 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.621059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.629227 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.629198    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:14:34.632416 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.632390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvtp\" (UniqueName: \"kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-gv224\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.633196 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.633181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.635715 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.635697    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 27 21:14:34.635824 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.635715    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 27 21:14:34.643674 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.643654    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:14:34.649532 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.649513    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:34.719890 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.719405    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.719890 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.719478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.719890 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.719540    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxll\" (UniqueName: \"kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.719890 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:34.719830    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-070ff-predictor-serving-cert: secret "success-200-isvc-070ff-predictor-serving-cert" not found
Jul 27 21:14:34.720421 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:34.719918    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls podName:c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e nodeName:}" failed. No retries permitted until 2026-07-27 21:14:35.21989406 +0000 UTC m=+638.890385703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls") pod "success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" (UID: "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e") : secret "success-200-isvc-070ff-predictor-serving-cert" not found
Jul 27 21:14:34.720947 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.720919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.793898 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.793861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxll\" (UniqueName: \"kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:34.820844 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.820806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.821034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.820867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp5x\" (UniqueName: \"kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.821034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.820890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.821034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.820913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.821034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.821006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.846133 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.846104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:14:34.846237 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:14:34.846143    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0084728c_27c2_4278_9c64_5430287c1def.slice/crio-282bfbb63f29ea0fcdd7fab0d7ca69fc0e279cabc4f2cab9fb839f010eb7b2ea WatchSource:0}: Error finding container 282bfbb63f29ea0fcdd7fab0d7ca69fc0e279cabc4f2cab9fb839f010eb7b2ea: Status 404 returned error can't find the container with id 282bfbb63f29ea0fcdd7fab0d7ca69fc0e279cabc4f2cab9fb839f010eb7b2ea
Jul 27 21:14:34.922441 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.922692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.922692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp5x\" (UniqueName: \"kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.922692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.922692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.922933 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:34.922688    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 27 21:14:34.922933 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:34.922768    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls podName:fc01c2b2-0d3d-40dd-9f49-1ea024627a25 nodeName:}" failed. No retries permitted until 2026-07-27 21:14:35.422733884 +0000 UTC m=+639.093225514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-km4vt" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 27 21:14:34.922933 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.922839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.923166 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.923146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.923330 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.923307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:34.938241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:34.938203    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp5x\" (UniqueName: \"kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:35.012480 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.012446    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:14:35.016460 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.016433    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.019229 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.019205    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 27 21:14:35.019229 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.019225    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 27 21:14:35.028868 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.028833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:14:35.124985 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.124905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.124985 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.124953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.125182 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.124985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.125182 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.125047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vmw\" (UniqueName: \"kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.125182 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.125064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vmw\" (UniqueName: \"kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.226685 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:35.226663    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 27 21:14:35.226741 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.226654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:35.226741 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:35.226733    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls podName:4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc nodeName:}" failed. No retries permitted until 2026-07-27 21:14:35.726714289 +0000 UTC m=+639.397205915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls") pod "isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 27 21:14:35.226891 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:35.226738    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-070ff-predictor-serving-cert: secret "success-200-isvc-070ff-predictor-serving-cert" not found
Jul 27 21:14:35.226891 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:14:35.226821    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls podName:c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e nodeName:}" failed. No retries permitted until 2026-07-27 21:14:36.226803572 +0000 UTC m=+639.897295199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls") pod "success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" (UID: "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e") : secret "success-200-isvc-070ff-predictor-serving-cert" not found
Jul 27 21:14:35.227152 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.227127    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.227253 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.227217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.227297 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.227217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.238340 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.238314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vmw\" (UniqueName: \"kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.427935 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.427842    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:35.430242 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.430222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-km4vt\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:35.550382 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.550342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:14:35.596556 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.596513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerStarted","Data":"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"}
Jul 27 21:14:35.596717 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.596588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerStarted","Data":"282bfbb63f29ea0fcdd7fab0d7ca69fc0e279cabc4f2cab9fb839f010eb7b2ea"}
Jul 27 21:14:35.690359 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.690334    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:14:35.692255 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:14:35.692223    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc01c2b2_0d3d_40dd_9f49_1ea024627a25.slice/crio-2c0fc4842a9458f1dfada9449bedfb240be2f01340771ef1783b6c70b1b232d5 WatchSource:0}: Error finding container 2c0fc4842a9458f1dfada9449bedfb240be2f01340771ef1783b6c70b1b232d5: Status 404 returned error can't find the container with id 2c0fc4842a9458f1dfada9449bedfb240be2f01340771ef1783b6c70b1b232d5
Jul 27 21:14:35.731137 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.731107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.733398 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.733369    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-bggx2\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:35.929339 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:35.929304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:36.069391 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.069365    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:14:36.071726 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:14:36.071701    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4400b7e1_c34f_47e2_b9cf_a55f0f73e9fc.slice/crio-424dd4833635a98b2a103fe2ff024fa477c30d238f6f7a2bfb58aefda587858f WatchSource:0}: Error finding container 424dd4833635a98b2a103fe2ff024fa477c30d238f6f7a2bfb58aefda587858f: Status 404 returned error can't find the container with id 424dd4833635a98b2a103fe2ff024fa477c30d238f6f7a2bfb58aefda587858f
Jul 27 21:14:36.236548 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.236451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:36.239049 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.239014    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"success-200-isvc-070ff-predictor-6488bc79fc-4ksmq\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") " pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:36.319486 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.319434    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:36.453347 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.453320    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:14:36.455402 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:14:36.455370    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93187fa_dfa6_4ab5_a0fc_8d66d8f34e1e.slice/crio-acbe2da59116c08510f00097775bb69b39b408fa7e1ca10aa74b59a770cc1b03 WatchSource:0}: Error finding container acbe2da59116c08510f00097775bb69b39b408fa7e1ca10aa74b59a770cc1b03: Status 404 returned error can't find the container with id acbe2da59116c08510f00097775bb69b39b408fa7e1ca10aa74b59a770cc1b03
Jul 27 21:14:36.605025 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.604989    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerStarted","Data":"acbe2da59116c08510f00097775bb69b39b408fa7e1ca10aa74b59a770cc1b03"}
Jul 27 21:14:36.606506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.606478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerStarted","Data":"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"}
Jul 27 21:14:36.606618 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.606539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerStarted","Data":"2c0fc4842a9458f1dfada9449bedfb240be2f01340771ef1783b6c70b1b232d5"}
Jul 27 21:14:36.608077 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.608043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerStarted","Data":"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"}
Jul 27 21:14:36.608184 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:36.608102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerStarted","Data":"424dd4833635a98b2a103fe2ff024fa477c30d238f6f7a2bfb58aefda587858f"}
Jul 27 21:14:39.630303 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:39.630261    2555 generic.go:356] "Generic (PLEG): container finished" podID="0084728c-27c2-4278-9c64-5430287c1def" containerID="4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393" exitCode=0
Jul 27 21:14:39.630788 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:39.630395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerDied","Data":"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"}
Jul 27 21:14:41.644784 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:41.644600    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerID="1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625" exitCode=0
Jul 27 21:14:41.644784 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:41.644692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerDied","Data":"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"}
Jul 27 21:14:41.649935 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:41.648940    2555 generic.go:356] "Generic (PLEG): container finished" podID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerID="0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e" exitCode=0
Jul 27 21:14:41.649935 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:41.649101    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerDied","Data":"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"}
Jul 27 21:14:54.747517 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:54.747421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerStarted","Data":"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"}
Jul 27 21:14:54.758966 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:54.758928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerStarted","Data":"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"}
Jul 27 21:14:54.762351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:54.762255    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerStarted","Data":"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"}
Jul 27 21:14:57.790837 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.790054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerStarted","Data":"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"}
Jul 27 21:14:57.790837 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.790128    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:57.790837 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.790158    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:14:57.791814 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.791683    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:14:57.794686 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.794658    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:57.794835 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.794820    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:14:57.795813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.795713    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:14:57.812971 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.812920    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podStartSLOduration=2.849209166 podStartE2EDuration="23.81290426s" totalImagesPullingTime="17.253813725s" totalInitContainerRuntime="3.709881369s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:57.810919217 +0000 UTC m=+661.481410901" watchObservedRunningTime="2026-07-27 21:14:57.81290426 +0000 UTC m=+661.483395906"
Jul 27 21:14:57.833248 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:57.833187    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podStartSLOduration=4.925700767 podStartE2EDuration="23.833163299s" totalImagesPullingTime="15.213144827s" totalInitContainerRuntime="3.694317705s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:57.83018033 +0000 UTC m=+661.500672001" watchObservedRunningTime="2026-07-27 21:14:57.833163299 +0000 UTC m=+661.503654945"
Jul 27 21:14:58.801174 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.801138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerStarted","Data":"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"}
Jul 27 21:14:58.801652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.801216    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:58.801652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.801378    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:14:58.802923 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.802888    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:14:58.803533 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.803506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerStarted","Data":"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"}
Jul 27 21:14:58.803708 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.803683    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:14:58.803857 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.803791    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:14:58.820210 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:58.820163    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podStartSLOduration=4.259625647 podStartE2EDuration="24.820147661s" totalImagesPullingTime="20.560522014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:58.818342118 +0000 UTC m=+662.488833787" watchObservedRunningTime="2026-07-27 21:14:58.820147661 +0000 UTC m=+662.490639380"
Jul 27 21:14:59.810795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:59.810631    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:14:59.810795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:14:59.810632    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:03.809326 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:03.809296    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:15:03.810120 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:03.810084    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:04.814859 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:04.814831    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:15:04.815529 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:04.815495    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:04.815806 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:04.815789    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:15:04.816277 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:04.816249    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:13.810644 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:13.810602    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:14.815827 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.815790    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:14.816202 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.816173    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:14.899033 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.898955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerStarted","Data":"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"}
Jul 27 21:15:14.899033 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.899021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerStarted","Data":"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"}
Jul 27 21:15:14.899299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.899281    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:15:14.899353 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.899327    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:15:14.900634 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.900610    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:14.919432 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:14.919373    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podStartSLOduration=4.438298543 podStartE2EDuration="40.919353492s" totalImagesPullingTime="32.935301746s" totalInitContainerRuntime="3.545753203s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:14.917472443 +0000 UTC m=+678.587964126" watchObservedRunningTime="2026-07-27 21:15:14.919353492 +0000 UTC m=+678.589845141"
Jul 27 21:15:15.903107 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:15.903067    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:20.908120 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:20.908089    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:15:20.908606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:20.908574    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:23.810536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:23.810490    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:24.815412 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:24.815365    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:24.816925 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:24.816902    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:30.909415 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:30.909368    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:33.810141 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:33.810103    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:34.815552 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:34.815454    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:34.816663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:34.816636    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:40.908803 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:40.908743    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:43.810283 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:43.810239    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:44.816233 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:44.816183    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:44.816627 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:44.816188    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:15:50.909002 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:50.908913    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:15:53.810723 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:53.810677    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:15:54.815399 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:54.815359    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:15:54.816900 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:15:54.816877    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:16:00.909357 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:00.909311    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:16:03.810178 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:03.810135    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 27 21:16:04.815631 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:04.815588    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:16:10.908654 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:10.908613    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:16:13.810907 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:13.810875    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:16:14.816369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:14.816315    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:16:20.908734 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:20.908695    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 27 21:16:24.588155 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.588120    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:16:24.588638 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.588385    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" containerID="cri-o://f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138" gracePeriod=30
Jul 27 21:16:24.588638 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.588413    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kube-rbac-proxy" containerID="cri-o://e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711" gracePeriod=30
Jul 27 21:16:24.633363 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.633330    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:16:24.636158 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.636140    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.638847 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.638829    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d355e-predictor-serving-cert\""
Jul 27 21:16:24.638933 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.638867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d355e-kube-rbac-proxy-sar-config\""
Jul 27 21:16:24.649976 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.649947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:16:24.767132 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.767090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.767289 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.767153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2csx\" (UniqueName: \"kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.767289 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.767227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.811480 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.811431    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 27 21:16:24.816202 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.816174    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:16:24.816461 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.816431    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 27 21:16:24.868374 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.868293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.868504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.868394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.868504 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:24.868426    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d355e-predictor-serving-cert: secret "success-200-isvc-d355e-predictor-serving-cert" not found
Jul 27 21:16:24.868504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.868436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2csx\" (UniqueName: \"kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.868504 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:24.868492    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls podName:bc456f43-7c87-4e48-9c2a-118520ab8903 nodeName:}" failed. No retries permitted until 2026-07-27 21:16:25.368469671 +0000 UTC m=+749.038961313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls") pod "success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" (UID: "bc456f43-7c87-4e48-9c2a-118520ab8903") : secret "success-200-isvc-d355e-predictor-serving-cert" not found
Jul 27 21:16:24.869065 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.869043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:24.879493 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:24.879464    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2csx\" (UniqueName: \"kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:25.229345 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.228989    2555 generic.go:356] "Generic (PLEG): container finished" podID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerID="e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711" exitCode=2
Jul 27 21:16:25.229345 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.229167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerDied","Data":"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"}
Jul 27 21:16:25.372654 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.372612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:25.375145 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.375110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") pod \"success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") " pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:25.546778 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.546651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:25.684087 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:25.684057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:16:25.685620 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:16:25.685596    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc456f43_7c87_4e48_9c2a_118520ab8903.slice/crio-a87bfce59baa43212edbb05491fe953fb07a01315d1d377cf71f862cc4e020ba WatchSource:0}: Error finding container a87bfce59baa43212edbb05491fe953fb07a01315d1d377cf71f862cc4e020ba: Status 404 returned error can't find the container with id a87bfce59baa43212edbb05491fe953fb07a01315d1d377cf71f862cc4e020ba
Jul 27 21:16:26.235927 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:26.235892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerStarted","Data":"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"}
Jul 27 21:16:26.236104 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:26.235954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerStarted","Data":"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"}
Jul 27 21:16:26.236104 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:26.235982    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:26.236104 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:26.236001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerStarted","Data":"a87bfce59baa43212edbb05491fe953fb07a01315d1d377cf71f862cc4e020ba"}
Jul 27 21:16:26.255515 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:26.255463    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podStartSLOduration=2.255445707 podStartE2EDuration="2.255445707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:26.253113747 +0000 UTC m=+749.923605393" watchObservedRunningTime="2026-07-27 21:16:26.255445707 +0000 UTC m=+749.925937354"
Jul 27 21:16:27.241048 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:27.241018    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:27.242101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:27.242079    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:16:27.929020 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:27.928997    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:16:28.097720 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.097685    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") pod \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") "
Jul 27 21:16:28.097907 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.097769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjxll\" (UniqueName: \"kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll\") pod \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") "
Jul 27 21:16:28.097907 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.097825    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config\") pod \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\" (UID: \"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e\") "
Jul 27 21:16:28.098227 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.098195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" (UID: "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:28.099943 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.099918    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll" (OuterVolumeSpecName: "kube-api-access-fjxll") pod "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" (UID: "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e"). InnerVolumeSpecName "kube-api-access-fjxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:16:28.099943 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.099938    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" (UID: "c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:16:28.198738 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.198703    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.198738 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.198730    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxll\" (UniqueName: \"kubernetes.io/projected/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-api-access-fjxll\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.198738 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.198740    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.247372 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.247336    2555 generic.go:356] "Generic (PLEG): container finished" podID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerID="f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138" exitCode=0
Jul 27 21:16:28.247799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.247392    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"
Jul 27 21:16:28.247799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.247421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerDied","Data":"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"}
Jul 27 21:16:28.247799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.247486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq" event={"ID":"c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e","Type":"ContainerDied","Data":"acbe2da59116c08510f00097775bb69b39b408fa7e1ca10aa74b59a770cc1b03"}
Jul 27 21:16:28.247799 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.247523    2555 scope.go:117] "RemoveContainer" containerID="e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"
Jul 27 21:16:28.248094 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.248067    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:16:28.256641 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.256622    2555 scope.go:117] "RemoveContainer" containerID="f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"
Jul 27 21:16:28.263844 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.263828    2555 scope.go:117] "RemoveContainer" containerID="e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"
Jul 27 21:16:28.264119 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:28.264099    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711\": container with ID starting with e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711 not found: ID does not exist" containerID="e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"
Jul 27 21:16:28.264168 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.264127    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711"} err="failed to get container status \"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711\": rpc error: code = NotFound desc = could not find container \"e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711\": container with ID starting with e05b4f07b4a0cd2ff8d4e03efbd85b0c43a72ee8609732b913524bd4455c1711 not found: ID does not exist"
Jul 27 21:16:28.264168 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.264145    2555 scope.go:117] "RemoveContainer" containerID="f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"
Jul 27 21:16:28.264357 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:28.264343    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138\": container with ID starting with f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138 not found: ID does not exist" containerID="f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"
Jul 27 21:16:28.264402 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.264363    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138"} err="failed to get container status \"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138\": rpc error: code = NotFound desc = could not find container \"f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138\": container with ID starting with f5ff75775e95287f1c138114e60dde22b7e3c1060cde27643f97b83c1352d138 not found: ID does not exist"
Jul 27 21:16:28.268745 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.268724    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:16:28.272682 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.272662    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-070ff-predictor-6488bc79fc-4ksmq"]
Jul 27 21:16:28.858231 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:28.858187    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" path="/var/lib/kubelet/pods/c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e/volumes"
Jul 27 21:16:30.909497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:30.909470    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:16:33.253055 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:33.253025    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:16:33.253546 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:33.253519    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:16:43.254271 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:43.254227    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:16:53.254066 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:53.254024    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:16:54.457770 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.457714    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:16:54.458154 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.458040    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" containerID="cri-o://082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0" gracePeriod=30
Jul 27 21:16:54.458154 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.458077    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy" containerID="cri-o://7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10" gracePeriod=30
Jul 27 21:16:54.514103 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.514072    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:16:54.514358 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.514330    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container" containerID="cri-o://406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660" gracePeriod=30
Jul 27 21:16:54.514358 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.514340    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kube-rbac-proxy" containerID="cri-o://0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd" gracePeriod=30
Jul 27 21:16:54.537628 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.537598    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:16:54.538032 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538019    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container"
Jul 27 21:16:54.538078 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538033    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container"
Jul 27 21:16:54.538078 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538043    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kube-rbac-proxy"
Jul 27 21:16:54.538078 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538049    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kube-rbac-proxy"
Jul 27 21:16:54.538167 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538101    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kube-rbac-proxy"
Jul 27 21:16:54.538167 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.538112    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93187fa-dfa6-4ab5-a0fc-8d66d8f34e1e" containerName="kserve-container"
Jul 27 21:16:54.540538 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.540521    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.544241 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.544221    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d13b2-predictor-serving-cert\""
Jul 27 21:16:54.544808 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.544792    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d13b2-kube-rbac-proxy-sar-config\""
Jul 27 21:16:54.560097 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.560069    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:16:54.610102 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.610067    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:16:54.610359 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.610332    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kube-rbac-proxy" containerID="cri-o://e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda" gracePeriod=30
Jul 27 21:16:54.610436 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.610314    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container" containerID="cri-o://adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1" gracePeriod=30
Jul 27 21:16:54.636557 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.636530    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.636696 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.636575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq277\" (UniqueName: \"kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.636696 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.636605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.737077 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.737034    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.737236 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.737146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.737236 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.737190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq277\" (UniqueName: \"kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.737719 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.737693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.739496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.739475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.745500 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.745474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq277\" (UniqueName: \"kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277\") pod \"success-200-isvc-d13b2-predictor-86c6db6b55-mf54z\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") " pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.811444 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.811397    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 27 21:16:54.815682 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.815657    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 27 21:16:54.851358 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.851329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:54.990932 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:54.990906    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:16:54.992706 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:16:54.992682    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb543262_741d_481b_b338_bb7d49c7774c.slice/crio-135c66911e5dfe5b738d1edcdb166a034f944e150754d6555d9fabbc2c8e0fa3 WatchSource:0}: Error finding container 135c66911e5dfe5b738d1edcdb166a034f944e150754d6555d9fabbc2c8e0fa3: Status 404 returned error can't find the container with id 135c66911e5dfe5b738d1edcdb166a034f944e150754d6555d9fabbc2c8e0fa3
Jul 27 21:16:55.375305 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.375273    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerID="e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda" exitCode=2
Jul 27 21:16:55.375483 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.375344    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerDied","Data":"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"}
Jul 27 21:16:55.377234 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.377215    2555 generic.go:356] "Generic (PLEG): container finished" podID="0084728c-27c2-4278-9c64-5430287c1def" containerID="0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd" exitCode=2
Jul 27 21:16:55.377324 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.377272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerDied","Data":"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"}
Jul 27 21:16:55.382806 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.382744    2555 generic.go:356] "Generic (PLEG): container finished" podID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerID="7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10" exitCode=2
Jul 27 21:16:55.382956 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.382860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerDied","Data":"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"}
Jul 27 21:16:55.385200 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.385168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerStarted","Data":"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"}
Jul 27 21:16:55.385334 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.385228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerStarted","Data":"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"}
Jul 27 21:16:55.385334 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.385258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerStarted","Data":"135c66911e5dfe5b738d1edcdb166a034f944e150754d6555d9fabbc2c8e0fa3"}
Jul 27 21:16:55.385861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.385784    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:55.385861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.385834    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:16:55.387409 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.387375    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:16:55.405611 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.405527    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podStartSLOduration=1.405510242 podStartE2EDuration="1.405510242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:55.404728632 +0000 UTC m=+779.075220280" watchObservedRunningTime="2026-07-27 21:16:55.405510242 +0000 UTC m=+779.076001887"
Jul 27 21:16:55.903833 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:55.903793    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 27 21:16:56.391401 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:56.391366    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:16:58.648604 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.648572    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:16:58.775328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775242    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert\") pod \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") "
Jul 27 21:16:58.775328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775298    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qp5x\" (UniqueName: \"kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x\") pod \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") "
Jul 27 21:16:58.775526 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775353    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config\") pod \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") "
Jul 27 21:16:58.775526 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775370    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") pod \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") "
Jul 27 21:16:58.775526 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location\") pod \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\" (UID: \"fc01c2b2-0d3d-40dd-9f49-1ea024627a25\") "
Jul 27 21:16:58.775725 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775696    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc01c2b2-0d3d-40dd-9f49-1ea024627a25" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:58.775874 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775687    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc01c2b2-0d3d-40dd-9f49-1ea024627a25" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:58.775874 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.775781    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc01c2b2-0d3d-40dd-9f49-1ea024627a25" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:16:58.777417 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.777394    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x" (OuterVolumeSpecName: "kube-api-access-2qp5x") pod "fc01c2b2-0d3d-40dd-9f49-1ea024627a25" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25"). InnerVolumeSpecName "kube-api-access-2qp5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:16:58.777504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.777436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc01c2b2-0d3d-40dd-9f49-1ea024627a25" (UID: "fc01c2b2-0d3d-40dd-9f49-1ea024627a25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:16:58.804210 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.804175    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 27 21:16:58.876817 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.876791    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kserve-provision-location\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:58.876817 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.876816    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-cabundle-cert\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:58.876998 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.876826    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qp5x\" (UniqueName: \"kubernetes.io/projected/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-api-access-2qp5x\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:58.876998 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.876836    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:58.876998 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:58.876846    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc01c2b2-0d3d-40dd-9f49-1ea024627a25-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:59.408243 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.408207    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerID="adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1" exitCode=0
Jul 27 21:16:59.408424 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.408272    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"
Jul 27 21:16:59.408424 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.408275    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerDied","Data":"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"}
Jul 27 21:16:59.408424 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.408339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt" event={"ID":"fc01c2b2-0d3d-40dd-9f49-1ea024627a25","Type":"ContainerDied","Data":"2c0fc4842a9458f1dfada9449bedfb240be2f01340771ef1783b6c70b1b232d5"}
Jul 27 21:16:59.408424 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.408378    2555 scope.go:117] "RemoveContainer" containerID="e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"
Jul 27 21:16:59.416693 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.416675    2555 scope.go:117] "RemoveContainer" containerID="adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"
Jul 27 21:16:59.424186 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.424170    2555 scope.go:117] "RemoveContainer" containerID="1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"
Jul 27 21:16:59.427675 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.427650    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:16:59.431362 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.431341    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-km4vt"]
Jul 27 21:16:59.432292 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.432273    2555 scope.go:117] "RemoveContainer" containerID="e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"
Jul 27 21:16:59.432562 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:59.432545    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda\": container with ID starting with e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda not found: ID does not exist" containerID="e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"
Jul 27 21:16:59.432639 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.432573    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda"} err="failed to get container status \"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda\": rpc error: code = NotFound desc = could not find container \"e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda\": container with ID starting with e92b2e1f623f02f08f9e0c3eb323f6c460af0524a2a0cd33d445a47392d45fda not found: ID does not exist"
Jul 27 21:16:59.432639 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.432597    2555 scope.go:117] "RemoveContainer" containerID="adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"
Jul 27 21:16:59.432859 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:59.432843    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1\": container with ID starting with adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1 not found: ID does not exist" containerID="adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"
Jul 27 21:16:59.432936 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.432867    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1"} err="failed to get container status \"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1\": rpc error: code = NotFound desc = could not find container \"adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1\": container with ID starting with adec8aa46f6e4bef7fb4c592ecfbf8acebd7936a0aca0eedc92fc000675271b1 not found: ID does not exist"
Jul 27 21:16:59.432936 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.432887    2555 scope.go:117] "RemoveContainer" containerID="1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"
Jul 27 21:16:59.433164 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:16:59.433144    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625\": container with ID starting with 1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625 not found: ID does not exist" containerID="1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"
Jul 27 21:16:59.433210 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.433169    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625"} err="failed to get container status \"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625\": rpc error: code = NotFound desc = could not find container \"1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625\": container with ID starting with 1bfec19de16eaddd067e3c275d1faf8018317b5027dc0ace48d81d9142287625 not found: ID does not exist"
Jul 27 21:16:59.811497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.811453    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 27 21:16:59.865606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.865581    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:16:59.985324 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985227    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location\") pod \"0084728c-27c2-4278-9c64-5430287c1def\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") "
Jul 27 21:16:59.985497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985326    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls\") pod \"0084728c-27c2-4278-9c64-5430287c1def\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") "
Jul 27 21:16:59.985497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert\") pod \"0084728c-27c2-4278-9c64-5430287c1def\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") "
Jul 27 21:16:59.985497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985424    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxvtp\" (UniqueName: \"kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp\") pod \"0084728c-27c2-4278-9c64-5430287c1def\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") "
Jul 27 21:16:59.985497 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config\") pod \"0084728c-27c2-4278-9c64-5430287c1def\" (UID: \"0084728c-27c2-4278-9c64-5430287c1def\") "
Jul 27 21:16:59.985784 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985597    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0084728c-27c2-4278-9c64-5430287c1def" (UID: "0084728c-27c2-4278-9c64-5430287c1def"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:16:59.985906 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0084728c-27c2-4278-9c64-5430287c1def" (UID: "0084728c-27c2-4278-9c64-5430287c1def"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:59.985970 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.985905    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0084728c-27c2-4278-9c64-5430287c1def" (UID: "0084728c-27c2-4278-9c64-5430287c1def"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:59.986328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.986239    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-cabundle-cert\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:59.986328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.986265    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0084728c-27c2-4278-9c64-5430287c1def-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:59.986328 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.986280    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0084728c-27c2-4278-9c64-5430287c1def-kserve-provision-location\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:16:59.987743 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.987718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0084728c-27c2-4278-9c64-5430287c1def" (UID: "0084728c-27c2-4278-9c64-5430287c1def"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:16:59.987978 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:16:59.987956    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp" (OuterVolumeSpecName: "kube-api-access-nxvtp") pod "0084728c-27c2-4278-9c64-5430287c1def" (UID: "0084728c-27c2-4278-9c64-5430287c1def"). InnerVolumeSpecName "kube-api-access-nxvtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:17:00.087062 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.087017    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0084728c-27c2-4278-9c64-5430287c1def-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.087062 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.087062    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvtp\" (UniqueName: \"kubernetes.io/projected/0084728c-27c2-4278-9c64-5430287c1def-kube-api-access-nxvtp\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.415391 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.415361    2555 generic.go:356] "Generic (PLEG): container finished" podID="0084728c-27c2-4278-9c64-5430287c1def" containerID="406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660" exitCode=0
Jul 27 21:17:00.415545 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.415431    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"
Jul 27 21:17:00.415545 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.415445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerDied","Data":"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"}
Jul 27 21:17:00.415545 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.415511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224" event={"ID":"0084728c-27c2-4278-9c64-5430287c1def","Type":"ContainerDied","Data":"282bfbb63f29ea0fcdd7fab0d7ca69fc0e279cabc4f2cab9fb839f010eb7b2ea"}
Jul 27 21:17:00.415643 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.415548    2555 scope.go:117] "RemoveContainer" containerID="0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"
Jul 27 21:17:00.450856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.450822    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:17:00.455180 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.455152    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-gv224"]
Jul 27 21:17:00.469581 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.469561    2555 scope.go:117] "RemoveContainer" containerID="406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"
Jul 27 21:17:00.477187 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.477169    2555 scope.go:117] "RemoveContainer" containerID="4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"
Jul 27 21:17:00.487057 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.487032    2555 scope.go:117] "RemoveContainer" containerID="0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"
Jul 27 21:17:00.487437 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:00.487408    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd\": container with ID starting with 0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd not found: ID does not exist" containerID="0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"
Jul 27 21:17:00.487537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.487450    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd"} err="failed to get container status \"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd\": rpc error: code = NotFound desc = could not find container \"0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd\": container with ID starting with 0f9fe84b2b9ebc6d960a6e6b97036e36f9e39f8b51e7c2f4f0ab99330519bbfd not found: ID does not exist"
Jul 27 21:17:00.487537 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.487478    2555 scope.go:117] "RemoveContainer" containerID="406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"
Jul 27 21:17:00.487788 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:00.487766    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660\": container with ID starting with 406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660 not found: ID does not exist" containerID="406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"
Jul 27 21:17:00.487869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.487791    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660"} err="failed to get container status \"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660\": rpc error: code = NotFound desc = could not find container \"406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660\": container with ID starting with 406ea7266d968d842bcaafa101e83103c86966cab3b0c28ecf111a07d5a97660 not found: ID does not exist"
Jul 27 21:17:00.487869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.487806    2555 scope.go:117] "RemoveContainer" containerID="4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"
Jul 27 21:17:00.488161 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:00.488133    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393\": container with ID starting with 4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393 not found: ID does not exist" containerID="4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"
Jul 27 21:17:00.488228 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.488171    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393"} err="failed to get container status \"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393\": rpc error: code = NotFound desc = could not find container \"4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393\": container with ID starting with 4f3fff72cef6611a1a0c6211e17fcdd372a480c10904702a8b29e9d349756393 not found: ID does not exist"
Jul 27 21:17:00.596443 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.596420    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:17:00.692101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692016    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") pod \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") "
Jul 27 21:17:00.692101 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692082    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8vmw\" (UniqueName: \"kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw\") pod \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") "
Jul 27 21:17:00.692306 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692131    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location\") pod \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") "
Jul 27 21:17:00.692306 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert\") pod \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") "
Jul 27 21:17:00.692306 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692197    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config\") pod \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\" (UID: \"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc\") "
Jul 27 21:17:00.692502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692446    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:17:00.692580 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692556    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:17:00.692627 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.692584    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:17:00.694179 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.694161    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw" (OuterVolumeSpecName: "kube-api-access-k8vmw") pod "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc"). InnerVolumeSpecName "kube-api-access-k8vmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:17:00.694414 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.694395    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" (UID: "4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:17:00.793085 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.793047    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.793085 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.793079    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vmw\" (UniqueName: \"kubernetes.io/projected/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-api-access-k8vmw\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.793287 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.793095    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kserve-provision-location\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.793287 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.793107    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-cabundle-cert\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.793287 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.793119    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.858512 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.858480    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0084728c-27c2-4278-9c64-5430287c1def" path="/var/lib/kubelet/pods/0084728c-27c2-4278-9c64-5430287c1def/volumes"
Jul 27 21:17:00.859004 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:00.858990    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" path="/var/lib/kubelet/pods/fc01c2b2-0d3d-40dd-9f49-1ea024627a25/volumes"
Jul 27 21:17:01.397510 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.397479    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:17:01.397977 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.397939    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:17:01.422569 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.422537    2555 generic.go:356] "Generic (PLEG): container finished" podID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerID="082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0" exitCode=0
Jul 27 21:17:01.422729 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.422595    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"
Jul 27 21:17:01.422729 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.422601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerDied","Data":"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"}
Jul 27 21:17:01.422729 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.422647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2" event={"ID":"4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc","Type":"ContainerDied","Data":"424dd4833635a98b2a103fe2ff024fa477c30d238f6f7a2bfb58aefda587858f"}
Jul 27 21:17:01.422729 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.422674    2555 scope.go:117] "RemoveContainer" containerID="7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"
Jul 27 21:17:01.430832 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.430805    2555 scope.go:117] "RemoveContainer" containerID="082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"
Jul 27 21:17:01.438685 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.438664    2555 scope.go:117] "RemoveContainer" containerID="0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"
Jul 27 21:17:01.440521 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.440489    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:17:01.442865 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.442846    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-bggx2"]
Jul 27 21:17:01.446366 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.446352    2555 scope.go:117] "RemoveContainer" containerID="7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"
Jul 27 21:17:01.446644 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:01.446625    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10\": container with ID starting with 7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10 not found: ID does not exist" containerID="7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"
Jul 27 21:17:01.446699 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.446653    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10"} err="failed to get container status \"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10\": rpc error: code = NotFound desc = could not find container \"7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10\": container with ID starting with 7cf6df1a4e786d18c4e3ca6362b450e041a2191bf53f17692c29e4c9b03d7b10 not found: ID does not exist"
Jul 27 21:17:01.446699 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.446671    2555 scope.go:117] "RemoveContainer" containerID="082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"
Jul 27 21:17:01.446933 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:01.446917    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0\": container with ID starting with 082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0 not found: ID does not exist" containerID="082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"
Jul 27 21:17:01.446984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.446939    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0"} err="failed to get container status \"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0\": rpc error: code = NotFound desc = could not find container \"082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0\": container with ID starting with 082a7078923ca66cab35567beabe1735cd8094116a654fc50a5b6ee451cb44e0 not found: ID does not exist"
Jul 27 21:17:01.446984 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.446955    2555 scope.go:117] "RemoveContainer" containerID="0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"
Jul 27 21:17:01.447179 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:17:01.447165    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e\": container with ID starting with 0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e not found: ID does not exist" containerID="0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"
Jul 27 21:17:01.447222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:01.447184    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e"} err="failed to get container status \"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e\": rpc error: code = NotFound desc = could not find container \"0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e\": container with ID starting with 0c151dfa4fb3729321e482c25ac2ddf02d82e53e585cb9f1d38ec7492df2eb4e not found: ID does not exist"
Jul 27 21:17:02.858310 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:02.858277    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" path="/var/lib/kubelet/pods/4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc/volumes"
Jul 27 21:17:03.253634 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:03.253545    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 27 21:17:11.398625 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:11.398585    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:17:13.254842 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:13.254811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:17:21.398152 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:21.398065    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:17:31.397950 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:31.397908    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:17:41.398900 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:17:41.398864    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:18:56.940938 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:18:56.940905    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache]"
Jul 27 21:25:39.443441 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.443408    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:25:39.445885 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.443644    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container" containerID="cri-o://de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797" gracePeriod=30
Jul 27 21:25:39.445885 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.443723    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kube-rbac-proxy" containerID="cri-o://a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f" gracePeriod=30
Jul 27 21:25:39.584692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.584660    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:25:39.585160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585131    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container"
Jul 27 21:25:39.585160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585154    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585167    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585175    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585194    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585202    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585247    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="storage-initializer"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585256    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="storage-initializer"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585276    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="storage-initializer"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585284    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="storage-initializer"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585294    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585302    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585310    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585316    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585333    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="storage-initializer"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585338    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="storage-initializer"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585343    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585348    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585422    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585431    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kube-rbac-proxy"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585437    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0084728c-27c2-4278-9c64-5430287c1def" containerName="kserve-container"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585443    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kserve-container"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585450    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4400b7e1-c34f-47e2-b9cf-a55f0f73e9fc" containerName="kserve-container"
Jul 27 21:25:39.585652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.585456    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc01c2b2-0d3d-40dd-9f49-1ea024627a25" containerName="kube-rbac-proxy"
Jul 27 21:25:39.588437 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.588420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.591166 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.591142    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb683-kube-rbac-proxy-sar-config\""
Jul 27 21:25:39.591306 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.591142    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb683-predictor-serving-cert\""
Jul 27 21:25:39.599431 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.599404    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:25:39.610213 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.610180    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerID="a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f" exitCode=2
Jul 27 21:25:39.610366 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.610248    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerDied","Data":"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"}
Jul 27 21:25:39.617816 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.617782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.617963 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.617933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszq5\" (UniqueName: \"kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.618017 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.617973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.718504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.718404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszq5\" (UniqueName: \"kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.718504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.718455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.718504 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.718475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.719114 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.719089    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.720960 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.720943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.729260 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.729232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszq5\" (UniqueName: \"kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5\") pod \"success-200-isvc-cb683-predictor-5944ff6487-6sbjs\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") " pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:39.899141 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:39.899096    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:40.034619 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.034595    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:25:40.037001 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:25:40.036977    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca42de00_87a9_496e_b072_944b45209902.slice/crio-6ed200ea963666391d2461c4b7efaffcf1023d5cb0db7e5947964d6565618635 WatchSource:0}: Error finding container 6ed200ea963666391d2461c4b7efaffcf1023d5cb0db7e5947964d6565618635: Status 404 returned error can't find the container with id 6ed200ea963666391d2461c4b7efaffcf1023d5cb0db7e5947964d6565618635
Jul 27 21:25:40.615883 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.615844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerStarted","Data":"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"}
Jul 27 21:25:40.616270 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.615908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerStarted","Data":"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"}
Jul 27 21:25:40.616270 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.615931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerStarted","Data":"6ed200ea963666391d2461c4b7efaffcf1023d5cb0db7e5947964d6565618635"}
Jul 27 21:25:40.616270 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.615961    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:40.637189 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:40.637139    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podStartSLOduration=1.637122924 podStartE2EDuration="1.637122924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:25:40.634683043 +0000 UTC m=+1304.305174694" watchObservedRunningTime="2026-07-27 21:25:40.637122924 +0000 UTC m=+1304.307614570"
Jul 27 21:25:41.620178 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:41.620144    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:41.621496 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:41.621471    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:25:42.624076 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.624028    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:25:42.792605 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.792572    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:25:42.844556 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.844452    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2csx\" (UniqueName: \"kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx\") pod \"bc456f43-7c87-4e48-9c2a-118520ab8903\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") "
Jul 27 21:25:42.844782 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.844566    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") pod \"bc456f43-7c87-4e48-9c2a-118520ab8903\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") "
Jul 27 21:25:42.844782 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.844598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config\") pod \"bc456f43-7c87-4e48-9c2a-118520ab8903\" (UID: \"bc456f43-7c87-4e48-9c2a-118520ab8903\") "
Jul 27 21:25:42.844989 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.844959    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc456f43-7c87-4e48-9c2a-118520ab8903" (UID: "bc456f43-7c87-4e48-9c2a-118520ab8903"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:42.846536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.846514    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx" (OuterVolumeSpecName: "kube-api-access-v2csx") pod "bc456f43-7c87-4e48-9c2a-118520ab8903" (UID: "bc456f43-7c87-4e48-9c2a-118520ab8903"). InnerVolumeSpecName "kube-api-access-v2csx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:42.846606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.846535    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc456f43-7c87-4e48-9c2a-118520ab8903" (UID: "bc456f43-7c87-4e48-9c2a-118520ab8903"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:42.945582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.945548    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc456f43-7c87-4e48-9c2a-118520ab8903-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:25:42.945582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.945575    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:25:42.945582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:42.945584    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2csx\" (UniqueName: \"kubernetes.io/projected/bc456f43-7c87-4e48-9c2a-118520ab8903-kube-api-access-v2csx\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:25:43.629362 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.629321    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerID="de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797" exitCode=0
Jul 27 21:25:43.629795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.629376    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"
Jul 27 21:25:43.629795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.629410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerDied","Data":"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"}
Jul 27 21:25:43.629795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.629472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n" event={"ID":"bc456f43-7c87-4e48-9c2a-118520ab8903","Type":"ContainerDied","Data":"a87bfce59baa43212edbb05491fe953fb07a01315d1d377cf71f862cc4e020ba"}
Jul 27 21:25:43.629795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.629501    2555 scope.go:117] "RemoveContainer" containerID="a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"
Jul 27 21:25:43.637890 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.637868    2555 scope.go:117] "RemoveContainer" containerID="de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"
Jul 27 21:25:43.645154 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.645115    2555 scope.go:117] "RemoveContainer" containerID="a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"
Jul 27 21:25:43.645511 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:25:43.645477    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f\": container with ID starting with a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f not found: ID does not exist" containerID="a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"
Jul 27 21:25:43.645598 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.645526    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f"} err="failed to get container status \"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f\": rpc error: code = NotFound desc = could not find container \"a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f\": container with ID starting with a2f4066882f5b7aa045bcd9edf14fbed286ddf2a0cf2ee68e1f88e956ef3562f not found: ID does not exist"
Jul 27 21:25:43.645598 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.645548    2555 scope.go:117] "RemoveContainer" containerID="de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"
Jul 27 21:25:43.645941 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:25:43.645915    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797\": container with ID starting with de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797 not found: ID does not exist" containerID="de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"
Jul 27 21:25:43.646036 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.645945    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797"} err="failed to get container status \"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797\": rpc error: code = NotFound desc = could not find container \"de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797\": container with ID starting with de87e08a9a510a5791160c38dc2e06f41502f41cce02675c5cd821c6d9e7f797 not found: ID does not exist"
Jul 27 21:25:43.647789 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.647770    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:25:43.650901 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:43.650879    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d355e-predictor-c7c6d9d4c-s9x8n"]
Jul 27 21:25:44.857578 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:44.857547    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" path="/var/lib/kubelet/pods/bc456f43-7c87-4e48-9c2a-118520ab8903/volumes"
Jul 27 21:25:47.628893 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:47.628862    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:25:47.629440 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:47.629414    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:25:57.629629 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:25:57.629589    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:26:07.629523 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:07.629486    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:26:09.253405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.253368    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:26:09.253904 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.253679    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" containerID="cri-o://be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7" gracePeriod=30
Jul 27 21:26:09.254156 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.254106    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kube-rbac-proxy" containerID="cri-o://b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d" gracePeriod=30
Jul 27 21:26:09.299055 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299023    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:26:09.299435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container"
Jul 27 21:26:09.299435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299426    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container"
Jul 27 21:26:09.299435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299435    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kube-rbac-proxy"
Jul 27 21:26:09.299435 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299441    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kube-rbac-proxy"
Jul 27 21:26:09.299694 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299513    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kserve-container"
Jul 27 21:26:09.299694 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.299522    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc456f43-7c87-4e48-9c2a-118520ab8903" containerName="kube-rbac-proxy"
Jul 27 21:26:09.303850 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.303832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.306832 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.306809    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a095-kube-rbac-proxy-sar-config\""
Jul 27 21:26:09.306928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.306833    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a095-predictor-serving-cert\""
Jul 27 21:26:09.312226 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.312203    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:26:09.370000 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.369970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.370165 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.370040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.370165 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.370067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xn98\" (UniqueName: \"kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.471191 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.471154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.471430 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.471314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.471430 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.471358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn98\" (UniqueName: \"kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.471561 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:09.471462    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0a095-predictor-serving-cert: secret "success-200-isvc-0a095-predictor-serving-cert" not found
Jul 27 21:26:09.471561 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:09.471560    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls podName:e4098812-fad8-4744-bf13-000e088a6126 nodeName:}" failed. No retries permitted until 2026-07-27 21:26:09.971539668 +0000 UTC m=+1333.642031294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls") pod "success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" (UID: "e4098812-fad8-4744-bf13-000e088a6126") : secret "success-200-isvc-0a095-predictor-serving-cert" not found
Jul 27 21:26:09.471944 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.471922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.482315 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.482279    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn98\" (UniqueName: \"kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.741706 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.741669    2555 generic.go:356] "Generic (PLEG): container finished" podID="cb543262-741d-481b-b338-bb7d49c7774c" containerID="b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d" exitCode=2
Jul 27 21:26:09.741884 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.741743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerDied","Data":"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"}
Jul 27 21:26:09.975897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.975846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:09.978200 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:09.978171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") pod \"success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") " pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:10.213977 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.213940    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:10.347825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.347803    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:26:10.349717 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:26:10.349693    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4098812_fad8_4744_bf13_000e088a6126.slice/crio-21f7cc0833b3fdb642d913aae8e073a937fedb3a2de073bd19878f0a4e80ef6a WatchSource:0}: Error finding container 21f7cc0833b3fdb642d913aae8e073a937fedb3a2de073bd19878f0a4e80ef6a: Status 404 returned error can't find the container with id 21f7cc0833b3fdb642d913aae8e073a937fedb3a2de073bd19878f0a4e80ef6a
Jul 27 21:26:10.747597 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.747484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerStarted","Data":"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"}
Jul 27 21:26:10.747597 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.747595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerStarted","Data":"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"}
Jul 27 21:26:10.747856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.747625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerStarted","Data":"21f7cc0833b3fdb642d913aae8e073a937fedb3a2de073bd19878f0a4e80ef6a"}
Jul 27 21:26:10.747856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.747656    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:10.747856 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.747683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:10.748970 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.748942    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:10.767916 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:10.767864    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podStartSLOduration=1.76784977 podStartE2EDuration="1.76784977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:10.766381562 +0000 UTC m=+1334.436873211" watchObservedRunningTime="2026-07-27 21:26:10.76784977 +0000 UTC m=+1334.438341499"
Jul 27 21:26:11.392001 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:11.391955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 27 21:26:11.398396 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:11.398362    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 27 21:26:11.751887 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:11.751788    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:12.605543 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.605516    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:26:12.701649 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.701570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq277\" (UniqueName: \"kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277\") pod \"cb543262-741d-481b-b338-bb7d49c7774c\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") "
Jul 27 21:26:12.701649 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.701617    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls\") pod \"cb543262-741d-481b-b338-bb7d49c7774c\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") "
Jul 27 21:26:12.701921 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.701657    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config\") pod \"cb543262-741d-481b-b338-bb7d49c7774c\" (UID: \"cb543262-741d-481b-b338-bb7d49c7774c\") "
Jul 27 21:26:12.702077 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.702051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb543262-741d-481b-b338-bb7d49c7774c" (UID: "cb543262-741d-481b-b338-bb7d49c7774c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:12.703731 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.703705    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277" (OuterVolumeSpecName: "kube-api-access-sq277") pod "cb543262-741d-481b-b338-bb7d49c7774c" (UID: "cb543262-741d-481b-b338-bb7d49c7774c"). InnerVolumeSpecName "kube-api-access-sq277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:12.703825 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.703738    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb543262-741d-481b-b338-bb7d49c7774c" (UID: "cb543262-741d-481b-b338-bb7d49c7774c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:12.757400 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.757363    2555 generic.go:356] "Generic (PLEG): container finished" podID="cb543262-741d-481b-b338-bb7d49c7774c" containerID="be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7" exitCode=0
Jul 27 21:26:12.757558 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.757418    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"
Jul 27 21:26:12.757558 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.757416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerDied","Data":"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"}
Jul 27 21:26:12.757558 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.757477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z" event={"ID":"cb543262-741d-481b-b338-bb7d49c7774c","Type":"ContainerDied","Data":"135c66911e5dfe5b738d1edcdb166a034f944e150754d6555d9fabbc2c8e0fa3"}
Jul 27 21:26:12.757558 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.757502    2555 scope.go:117] "RemoveContainer" containerID="b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"
Jul 27 21:26:12.766506 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.766486    2555 scope.go:117] "RemoveContainer" containerID="be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"
Jul 27 21:26:12.773737 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.773718    2555 scope.go:117] "RemoveContainer" containerID="b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"
Jul 27 21:26:12.773992 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:12.773969    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d\": container with ID starting with b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d not found: ID does not exist" containerID="b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"
Jul 27 21:26:12.774051 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.774001    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d"} err="failed to get container status \"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d\": rpc error: code = NotFound desc = could not find container \"b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d\": container with ID starting with b9c782b8acaec427edcec35d7d15dc2b121f367d5aa59f8fb99eb1e03464684d not found: ID does not exist"
Jul 27 21:26:12.774051 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.774019    2555 scope.go:117] "RemoveContainer" containerID="be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"
Jul 27 21:26:12.774267 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:12.774248    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7\": container with ID starting with be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7 not found: ID does not exist" containerID="be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"
Jul 27 21:26:12.774320 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.774276    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7"} err="failed to get container status \"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7\": rpc error: code = NotFound desc = could not find container \"be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7\": container with ID starting with be1f4e739a5272e14414a0bdfc8ad52ae514489ab2035ce4709ab7f678aa7bf7 not found: ID does not exist"
Jul 27 21:26:12.778705 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.778682    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:26:12.784193 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.784159    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d13b2-predictor-86c6db6b55-mf54z"]
Jul 27 21:26:12.803073 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.803047    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq277\" (UniqueName: \"kubernetes.io/projected/cb543262-741d-481b-b338-bb7d49c7774c-kube-api-access-sq277\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.803073 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.803074    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb543262-741d-481b-b338-bb7d49c7774c-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.803223 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.803089    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb543262-741d-481b-b338-bb7d49c7774c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.858338 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:12.858306    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb543262-741d-481b-b338-bb7d49c7774c" path="/var/lib/kubelet/pods/cb543262-741d-481b-b338-bb7d49c7774c/volumes"
Jul 27 21:26:16.756248 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:16.756218    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:16.756712 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:16.756631    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:17.629478 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:17.629433    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 27 21:26:26.757106 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:26.757065    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:27.630137 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:27.630108    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:26:36.757274 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:36.757230    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:46.757672 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:46.757631    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 27 21:26:49.855704 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.855666    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:26:49.856160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.855920    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container" containerID="cri-o://8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328" gracePeriod=30
Jul 27 21:26:49.856160 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.855968    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kube-rbac-proxy" containerID="cri-o://783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb" gracePeriod=30
Jul 27 21:26:49.872163 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872138    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:26:49.872498 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872486    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container"
Jul 27 21:26:49.872498 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872498    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container"
Jul 27 21:26:49.872610 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872533    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kube-rbac-proxy"
Jul 27 21:26:49.872610 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872538    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kube-rbac-proxy"
Jul 27 21:26:49.872610 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872590    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kube-rbac-proxy"
Jul 27 21:26:49.872610 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.872599    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb543262-741d-481b-b338-bb7d49c7774c" containerName="kserve-container"
Jul 27 21:26:49.875841 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.875825    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:49.879406 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.879382    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e26c5-predictor-serving-cert\""
Jul 27 21:26:49.879515 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.879492    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e26c5-kube-rbac-proxy-sar-config\""
Jul 27 21:26:49.887519 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.887495    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:26:49.931866 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.931818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:49.932066 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.931993    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46m8\" (UniqueName: \"kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:49.932066 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:49.932037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.032929 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.032892    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.033123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.033050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46m8\" (UniqueName: \"kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.033123 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.033085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.033544 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.033521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.035509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.035487    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.042272 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.042251    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46m8\" (UniqueName: \"kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8\") pod \"success-200-isvc-e26c5-predictor-d5cd578f-9wk7v\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") " pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.187655 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.187565    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.320215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.320188    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:26:50.322677 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:26:50.322649    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f6f403_0fce_4bdf_9483_b6461d287931.slice/crio-d6358673bb17126260e5f778c0f33eb91cfb9dc40a453b17086f16375bf6cffc WatchSource:0}: Error finding container d6358673bb17126260e5f778c0f33eb91cfb9dc40a453b17086f16375bf6cffc: Status 404 returned error can't find the container with id d6358673bb17126260e5f778c0f33eb91cfb9dc40a453b17086f16375bf6cffc
Jul 27 21:26:50.927074 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.927036    2555 generic.go:356] "Generic (PLEG): container finished" podID="ca42de00-87a9-496e-b072-944b45209902" containerID="783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb" exitCode=2
Jul 27 21:26:50.927445 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.927109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerDied","Data":"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"}
Jul 27 21:26:50.928847 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.928827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerStarted","Data":"14b93a4772839e01779dd2edb04f4ceb00070da39d2f85732495ebd30f77c3a9"}
Jul 27 21:26:50.928955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.928876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerStarted","Data":"748a29c61a5650a58f86c1f06e6740e4bf2404229c81800a0ff681a4700341df"}
Jul 27 21:26:50.928955 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.928902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerStarted","Data":"d6358673bb17126260e5f778c0f33eb91cfb9dc40a453b17086f16375bf6cffc"}
Jul 27 21:26:50.929080 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.929064    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.929237 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.929220    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:50.930354 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.930330    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:26:50.948829 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:50.948789    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podStartSLOduration=1.9487771120000001 podStartE2EDuration="1.948777112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:50.946901108 +0000 UTC m=+1374.617392773" watchObservedRunningTime="2026-07-27 21:26:50.948777112 +0000 UTC m=+1374.619268753"
Jul 27 21:26:51.933208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:51.933171    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:26:52.625057 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:52.625012    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 27 21:26:53.306858 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.306834    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:26:53.360603 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.360512    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls\") pod \"ca42de00-87a9-496e-b072-944b45209902\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") "
Jul 27 21:26:53.360603 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.360551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dszq5\" (UniqueName: \"kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5\") pod \"ca42de00-87a9-496e-b072-944b45209902\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") "
Jul 27 21:26:53.360870 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.360609    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config\") pod \"ca42de00-87a9-496e-b072-944b45209902\" (UID: \"ca42de00-87a9-496e-b072-944b45209902\") "
Jul 27 21:26:53.360993 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.360968    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca42de00-87a9-496e-b072-944b45209902" (UID: "ca42de00-87a9-496e-b072-944b45209902"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:53.362572 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.362549    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca42de00-87a9-496e-b072-944b45209902" (UID: "ca42de00-87a9-496e-b072-944b45209902"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:53.362718 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.362698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5" (OuterVolumeSpecName: "kube-api-access-dszq5") pod "ca42de00-87a9-496e-b072-944b45209902" (UID: "ca42de00-87a9-496e-b072-944b45209902"). InnerVolumeSpecName "kube-api-access-dszq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:53.462096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.462061    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca42de00-87a9-496e-b072-944b45209902-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.462096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.462088    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszq5\" (UniqueName: \"kubernetes.io/projected/ca42de00-87a9-496e-b072-944b45209902-kube-api-access-dszq5\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.462096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.462097    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca42de00-87a9-496e-b072-944b45209902-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.943873 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.943842    2555 generic.go:356] "Generic (PLEG): container finished" podID="ca42de00-87a9-496e-b072-944b45209902" containerID="8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328" exitCode=0
Jul 27 21:26:53.944063 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.943894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerDied","Data":"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"}
Jul 27 21:26:53.944063 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.943939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs" event={"ID":"ca42de00-87a9-496e-b072-944b45209902","Type":"ContainerDied","Data":"6ed200ea963666391d2461c4b7efaffcf1023d5cb0db7e5947964d6565618635"}
Jul 27 21:26:53.944063 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.943896    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"
Jul 27 21:26:53.944063 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.943964    2555 scope.go:117] "RemoveContainer" containerID="783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"
Jul 27 21:26:53.952781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.952740    2555 scope.go:117] "RemoveContainer" containerID="8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"
Jul 27 21:26:53.960096 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.960076    2555 scope.go:117] "RemoveContainer" containerID="783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"
Jul 27 21:26:53.960347 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:53.960329    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb\": container with ID starting with 783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb not found: ID does not exist" containerID="783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"
Jul 27 21:26:53.960411 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.960356    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb"} err="failed to get container status \"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb\": rpc error: code = NotFound desc = could not find container \"783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb\": container with ID starting with 783630a489d7794207f3931127c169ae0b80871b43029ee7bd8717abccdb3dcb not found: ID does not exist"
Jul 27 21:26:53.960411 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.960375    2555 scope.go:117] "RemoveContainer" containerID="8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"
Jul 27 21:26:53.960573 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:26:53.960558    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328\": container with ID starting with 8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328 not found: ID does not exist" containerID="8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"
Jul 27 21:26:53.960616 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.960578    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328"} err="failed to get container status \"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328\": rpc error: code = NotFound desc = could not find container \"8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328\": container with ID starting with 8574bb73299ed60ede211798c98c6bf7e987bcf080521c763040dc4eda50d328 not found: ID does not exist"
Jul 27 21:26:53.965157 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.965137    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:26:53.969220 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:53.969195    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb683-predictor-5944ff6487-6sbjs"]
Jul 27 21:26:54.857531 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:54.857496    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca42de00-87a9-496e-b072-944b45209902" path="/var/lib/kubelet/pods/ca42de00-87a9-496e-b072-944b45209902/volumes"
Jul 27 21:26:56.757937 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:56.757904    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:26:56.937072 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:56.937045    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:26:56.937570 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:26:56.937538    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:27:06.937539 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:06.937500    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:27:16.937544 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:16.937507    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:27:19.499193 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.499164    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:27:19.499567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.499392    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container" containerID="cri-o://59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d" gracePeriod=30
Jul 27 21:27:19.499567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.499432    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kube-rbac-proxy" containerID="cri-o://d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc" gracePeriod=30
Jul 27 21:27:19.570296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570265    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:27:19.570654 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570642    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container"
Jul 27 21:27:19.570703 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570655    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container"
Jul 27 21:27:19.570703 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570666    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kube-rbac-proxy"
Jul 27 21:27:19.570703 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570672    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kube-rbac-proxy"
Jul 27 21:27:19.570817 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570736    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kube-rbac-proxy"
Jul 27 21:27:19.570817 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.570744    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca42de00-87a9-496e-b072-944b45209902" containerName="kserve-container"
Jul 27 21:27:19.575315 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.575293    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.578376 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.578240    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0f5d7-predictor-serving-cert\""
Jul 27 21:27:19.578376 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.578285    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0f5d7-kube-rbac-proxy-sar-config\""
Jul 27 21:27:19.585102 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.585078    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:27:19.684866 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.684802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.684866 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.684843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.685111 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.685064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rdn\" (UniqueName: \"kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.786317 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.786239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rdn\" (UniqueName: \"kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.786317 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.786275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.786317 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.786295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.786558 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:27:19.786419    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-serving-cert: secret "success-200-isvc-0f5d7-predictor-serving-cert" not found
Jul 27 21:27:19.786558 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:27:19.786494    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls podName:d4f550dd-99e5-4ddd-b84c-e89d1af94c2b nodeName:}" failed. No retries permitted until 2026-07-27 21:27:20.286472393 +0000 UTC m=+1403.956964037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls") pod "success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" (UID: "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b") : secret "success-200-isvc-0f5d7-predictor-serving-cert" not found
Jul 27 21:27:19.786933 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.786913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:19.797168 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:19.797147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rdn\" (UniqueName: \"kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:20.068727 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.068642    2555 generic.go:356] "Generic (PLEG): container finished" podID="e4098812-fad8-4744-bf13-000e088a6126" containerID="d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc" exitCode=2
Jul 27 21:27:20.068897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.068721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerDied","Data":"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"}
Jul 27 21:27:20.291163 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.291126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:20.293493 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.293470    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") pod \"success-200-isvc-0f5d7-predictor-69c54f994c-99rb7\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") " pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:20.487336 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.487299    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:20.614292 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:20.614265    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:27:20.616294 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:27:20.616264    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f550dd_99e5_4ddd_b84c_e89d1af94c2b.slice/crio-9e8960133e9752c1969b4ff9dd4729ef759806d0c88a7fcf2d3f46fe8fc8ca81 WatchSource:0}: Error finding container 9e8960133e9752c1969b4ff9dd4729ef759806d0c88a7fcf2d3f46fe8fc8ca81: Status 404 returned error can't find the container with id 9e8960133e9752c1969b4ff9dd4729ef759806d0c88a7fcf2d3f46fe8fc8ca81
Jul 27 21:27:21.075199 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.075165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerStarted","Data":"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"}
Jul 27 21:27:21.075410 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.075224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerStarted","Data":"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"}
Jul 27 21:27:21.075410 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.075250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerStarted","Data":"9e8960133e9752c1969b4ff9dd4729ef759806d0c88a7fcf2d3f46fe8fc8ca81"}
Jul 27 21:27:21.075410 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.075354    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:21.075410 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.075394    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:21.076475 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.076451    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:27:21.094606 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.094552    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podStartSLOduration=2.094534042 podStartE2EDuration="2.094534042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:21.092671892 +0000 UTC m=+1404.763163552" watchObservedRunningTime="2026-07-27 21:27:21.094534042 +0000 UTC m=+1404.765025690"
Jul 27 21:27:21.752841 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:21.752804    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 27 21:27:22.079639 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:22.079603    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:27:22.948464 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:22.948440    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:27:23.008997 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.008970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xn98\" (UniqueName: \"kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98\") pod \"e4098812-fad8-4744-bf13-000e088a6126\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") "
Jul 27 21:27:23.009149 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.009022    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config\") pod \"e4098812-fad8-4744-bf13-000e088a6126\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") "
Jul 27 21:27:23.009149 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.009061    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") pod \"e4098812-fad8-4744-bf13-000e088a6126\" (UID: \"e4098812-fad8-4744-bf13-000e088a6126\") "
Jul 27 21:27:23.009391 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.009368    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4098812-fad8-4744-bf13-000e088a6126" (UID: "e4098812-fad8-4744-bf13-000e088a6126"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:23.010976 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.010952    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98" (OuterVolumeSpecName: "kube-api-access-5xn98") pod "e4098812-fad8-4744-bf13-000e088a6126" (UID: "e4098812-fad8-4744-bf13-000e088a6126"). InnerVolumeSpecName "kube-api-access-5xn98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:23.011066 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.011037    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4098812-fad8-4744-bf13-000e088a6126" (UID: "e4098812-fad8-4744-bf13-000e088a6126"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:27:23.086463 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.086432    2555 generic.go:356] "Generic (PLEG): container finished" podID="e4098812-fad8-4744-bf13-000e088a6126" containerID="59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d" exitCode=0
Jul 27 21:27:23.086614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.086486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"
Jul 27 21:27:23.086614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.086507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerDied","Data":"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"}
Jul 27 21:27:23.086614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.086580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5" event={"ID":"e4098812-fad8-4744-bf13-000e088a6126","Type":"ContainerDied","Data":"21f7cc0833b3fdb642d913aae8e073a937fedb3a2de073bd19878f0a4e80ef6a"}
Jul 27 21:27:23.086802 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.086616    2555 scope.go:117] "RemoveContainer" containerID="d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"
Jul 27 21:27:23.095697 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.095681    2555 scope.go:117] "RemoveContainer" containerID="59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"
Jul 27 21:27:23.102489 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.102472    2555 scope.go:117] "RemoveContainer" containerID="d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"
Jul 27 21:27:23.102727 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:27:23.102709    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc\": container with ID starting with d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc not found: ID does not exist" containerID="d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"
Jul 27 21:27:23.102811 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.102735    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc"} err="failed to get container status \"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc\": rpc error: code = NotFound desc = could not find container \"d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc\": container with ID starting with d72b77b708152cf434388f1cded26866503ccab2524a9053de229b9de22551bc not found: ID does not exist"
Jul 27 21:27:23.103213 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.103200    2555 scope.go:117] "RemoveContainer" containerID="59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"
Jul 27 21:27:23.103436 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:27:23.103419    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d\": container with ID starting with 59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d not found: ID does not exist" containerID="59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"
Jul 27 21:27:23.103491 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.103441    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d"} err="failed to get container status \"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d\": rpc error: code = NotFound desc = could not find container \"59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d\": container with ID starting with 59311e6eb79b1141f72469a7bcec7e7a2c771f8cf683d5c382fdb42da86d485d not found: ID does not exist"
Jul 27 21:27:23.109463 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.109441    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:27:23.109781 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.109766    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xn98\" (UniqueName: \"kubernetes.io/projected/e4098812-fad8-4744-bf13-000e088a6126-kube-api-access-5xn98\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.109834 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.109786    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4098812-fad8-4744-bf13-000e088a6126-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.109834 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.109796    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4098812-fad8-4744-bf13-000e088a6126-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.113300 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:23.113275    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a095-predictor-7c5d79bf89-bfbs5"]
Jul 27 21:27:24.858198 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:24.858158    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4098812-fad8-4744-bf13-000e088a6126" path="/var/lib/kubelet/pods/e4098812-fad8-4744-bf13-000e088a6126/volumes"
Jul 27 21:27:26.937457 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:26.937424    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:27:27.090931 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:27.090900    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:27:27.091468 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:27.091438    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:27:36.938098 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:36.938069    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:27:37.092325 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:37.092289    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:27:47.092208 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:47.092167    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:27:57.091654 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:27:57.091612    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 27 21:28:07.091934 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:28:07.091902    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:36:04.645440 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.645404    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:36:04.646050 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.645626    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" containerID="cri-o://748a29c61a5650a58f86c1f06e6740e4bf2404229c81800a0ff681a4700341df" gracePeriod=30
Jul 27 21:36:04.646050 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.645675    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kube-rbac-proxy" containerID="cri-o://14b93a4772839e01779dd2edb04f4ceb00070da39d2f85732495ebd30f77c3a9" gracePeriod=30
Jul 27 21:36:04.703471 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703448    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:36:04.703816 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703803    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kube-rbac-proxy"
Jul 27 21:36:04.703861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703818    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kube-rbac-proxy"
Jul 27 21:36:04.703861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703836    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container"
Jul 27 21:36:04.703861 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703842    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container"
Jul 27 21:36:04.703958 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703892    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kserve-container"
Jul 27 21:36:04.703958 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.703904    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4098812-fad8-4744-bf13-000e088a6126" containerName="kube-rbac-proxy"
Jul 27 21:36:04.706887 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.706870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.709383 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.709362    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-61561-kube-rbac-proxy-sar-config\""
Jul 27 21:36:04.709467 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.709362    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-61561-predictor-serving-cert\""
Jul 27 21:36:04.716793 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.716774    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:36:04.773489 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.773464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.773604 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.773569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6mw\" (UniqueName: \"kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.773667 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.773603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.874224 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.874199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6mw\" (UniqueName: \"kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.874351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.874231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.874427 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:04.874412    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-61561-predictor-serving-cert: secret "success-200-isvc-61561-predictor-serving-cert" not found
Jul 27 21:36:04.874464 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.874428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.874498 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:04.874468    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls podName:ce42dd03-8598-4512-9162-bc0232e54d77 nodeName:}" failed. No retries permitted until 2026-07-27 21:36:05.374452126 +0000 UTC m=+1929.044943753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls") pod "success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" (UID: "ce42dd03-8598-4512-9162-bc0232e54d77") : secret "success-200-isvc-61561-predictor-serving-cert" not found
Jul 27 21:36:04.875062 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.875045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:04.883680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:04.883660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6mw\" (UniqueName: \"kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:05.273041 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.273012    2555 generic.go:356] "Generic (PLEG): container finished" podID="30f6f403-0fce-4bdf-9483-b6461d287931" containerID="14b93a4772839e01779dd2edb04f4ceb00070da39d2f85732495ebd30f77c3a9" exitCode=2
Jul 27 21:36:05.273224 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.273092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerDied","Data":"14b93a4772839e01779dd2edb04f4ceb00070da39d2f85732495ebd30f77c3a9"}
Jul 27 21:36:05.378987 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.378952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:05.381187 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.381169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") pod \"success-200-isvc-61561-predictor-6959dd4cf7-jjqv6\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") " pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:05.617452 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.617421    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:05.741389 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:05.741365    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:36:05.743194 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:36:05.743169    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce42dd03_8598_4512_9162_bc0232e54d77.slice/crio-def6335986b00718b7cef81435677e99116bb3c2020e6ef53bf2a1a8c4e0ec9e WatchSource:0}: Error finding container def6335986b00718b7cef81435677e99116bb3c2020e6ef53bf2a1a8c4e0ec9e: Status 404 returned error can't find the container with id def6335986b00718b7cef81435677e99116bb3c2020e6ef53bf2a1a8c4e0ec9e
Jul 27 21:36:06.279599 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.279519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerStarted","Data":"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"}
Jul 27 21:36:06.279599 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.279573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerStarted","Data":"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"}
Jul 27 21:36:06.279815 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.279605    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerStarted","Data":"def6335986b00718b7cef81435677e99116bb3c2020e6ef53bf2a1a8c4e0ec9e"}
Jul 27 21:36:06.279815 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.279671    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:06.299813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.299765    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podStartSLOduration=2.299733588 podStartE2EDuration="2.299733588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:06.2974165 +0000 UTC m=+1929.967908160" watchObservedRunningTime="2026-07-27 21:36:06.299733588 +0000 UTC m=+1929.970225235"
Jul 27 21:36:06.934200 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.934154    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 27 21:36:06.937557 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:06.937522    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 27 21:36:07.287949 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.287912    2555 generic.go:356] "Generic (PLEG): container finished" podID="30f6f403-0fce-4bdf-9483-b6461d287931" containerID="748a29c61a5650a58f86c1f06e6740e4bf2404229c81800a0ff681a4700341df" exitCode=0
Jul 27 21:36:07.288094 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.287994    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerDied","Data":"748a29c61a5650a58f86c1f06e6740e4bf2404229c81800a0ff681a4700341df"}
Jul 27 21:36:07.288408 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.288390    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:07.289386 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.289362    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:07.381388 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.381361    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:36:07.499009 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.498953    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls\") pod \"30f6f403-0fce-4bdf-9483-b6461d287931\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") "
Jul 27 21:36:07.499114 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.499020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l46m8\" (UniqueName: \"kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8\") pod \"30f6f403-0fce-4bdf-9483-b6461d287931\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") "
Jul 27 21:36:07.499114 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.499088    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config\") pod \"30f6f403-0fce-4bdf-9483-b6461d287931\" (UID: \"30f6f403-0fce-4bdf-9483-b6461d287931\") "
Jul 27 21:36:07.499410 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.499382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30f6f403-0fce-4bdf-9483-b6461d287931" (UID: "30f6f403-0fce-4bdf-9483-b6461d287931"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:07.500864 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.500844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30f6f403-0fce-4bdf-9483-b6461d287931" (UID: "30f6f403-0fce-4bdf-9483-b6461d287931"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:07.500938 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.500899    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8" (OuterVolumeSpecName: "kube-api-access-l46m8") pod "30f6f403-0fce-4bdf-9483-b6461d287931" (UID: "30f6f403-0fce-4bdf-9483-b6461d287931"). InnerVolumeSpecName "kube-api-access-l46m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:07.600351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.600332    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46m8\" (UniqueName: \"kubernetes.io/projected/30f6f403-0fce-4bdf-9483-b6461d287931-kube-api-access-l46m8\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:07.600351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.600351    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f6f403-0fce-4bdf-9483-b6461d287931-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:07.600473 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:07.600360    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f6f403-0fce-4bdf-9483-b6461d287931-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:08.293522 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.293496    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"
Jul 27 21:36:08.293910 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.293521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v" event={"ID":"30f6f403-0fce-4bdf-9483-b6461d287931","Type":"ContainerDied","Data":"d6358673bb17126260e5f778c0f33eb91cfb9dc40a453b17086f16375bf6cffc"}
Jul 27 21:36:08.293910 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.293598    2555 scope.go:117] "RemoveContainer" containerID="14b93a4772839e01779dd2edb04f4ceb00070da39d2f85732495ebd30f77c3a9"
Jul 27 21:36:08.294173 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.294146    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:08.302502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.302481    2555 scope.go:117] "RemoveContainer" containerID="748a29c61a5650a58f86c1f06e6740e4bf2404229c81800a0ff681a4700341df"
Jul 27 21:36:08.315345 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.315321    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:36:08.319771 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.319734    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e26c5-predictor-d5cd578f-9wk7v"]
Jul 27 21:36:08.858499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:08.858472    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" path="/var/lib/kubelet/pods/30f6f403-0fce-4bdf-9483-b6461d287931/volumes"
Jul 27 21:36:13.298740 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:13.298711    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:13.299329 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:13.299231    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:23.299291 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:23.299251    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:33.299692 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:33.299651    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:34.412734 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.412693    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:36:34.413204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.412985    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container" containerID="cri-o://7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a" gracePeriod=30
Jul 27 21:36:34.413204 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.413040    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kube-rbac-proxy" containerID="cri-o://62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0" gracePeriod=30
Jul 27 21:36:34.466116 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466089    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:36:34.466446 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466433    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container"
Jul 27 21:36:34.466492 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466448    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container"
Jul 27 21:36:34.466492 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466464    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kube-rbac-proxy"
Jul 27 21:36:34.466492 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466470    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kube-rbac-proxy"
Jul 27 21:36:34.466591 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466532    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kserve-container"
Jul 27 21:36:34.466591 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.466542    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f6f403-0fce-4bdf-9483-b6461d287931" containerName="kube-rbac-proxy"
Jul 27 21:36:34.469829 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.469811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.472525 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.472507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-beb6a-predictor-serving-cert\""
Jul 27 21:36:34.472622 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.472541    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-beb6a-kube-rbac-proxy-sar-config\""
Jul 27 21:36:34.480366 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.480347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:36:34.602275 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.602248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.602405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.602284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.602405 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.602346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l246r\" (UniqueName: \"kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.702813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.702723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.702813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.702779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.702988 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.702817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l246r\" (UniqueName: \"kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.702988 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:34.702943    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-serving-cert: secret "success-200-isvc-beb6a-predictor-serving-cert" not found
Jul 27 21:36:34.703069 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:34.703009    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls podName:a10c2012-9cac-49ee-9d83-fdccde059d06 nodeName:}" failed. No retries permitted until 2026-07-27 21:36:35.202991358 +0000 UTC m=+1958.873482984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls") pod "success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" (UID: "a10c2012-9cac-49ee-9d83-fdccde059d06") : secret "success-200-isvc-beb6a-predictor-serving-cert" not found
Jul 27 21:36:34.703319 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.703301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:34.712512 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:34.712492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l246r\" (UniqueName: \"kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:35.206296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:35.206265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:35.206462 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:35.206407    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-serving-cert: secret "success-200-isvc-beb6a-predictor-serving-cert" not found
Jul 27 21:36:35.206505 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:35.206485    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls podName:a10c2012-9cac-49ee-9d83-fdccde059d06 nodeName:}" failed. No retries permitted until 2026-07-27 21:36:36.206466971 +0000 UTC m=+1959.876958598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls") pod "success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" (UID: "a10c2012-9cac-49ee-9d83-fdccde059d06") : secret "success-200-isvc-beb6a-predictor-serving-cert" not found
Jul 27 21:36:35.408975 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:35.408942    2555 generic.go:356] "Generic (PLEG): container finished" podID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerID="62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0" exitCode=2
Jul 27 21:36:35.409131 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:35.409009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerDied","Data":"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"}
Jul 27 21:36:36.214215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.214178    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:36.216616 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.216593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"success-200-isvc-beb6a-predictor-d9b4c445f-kbknm\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") " pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:36.279467 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.279434    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:36.404568 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.404547    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:36:36.406541 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:36:36.406503    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10c2012_9cac_49ee_9d83_fdccde059d06.slice/crio-b467fecfccc3e82d66a402895b2cac1734afa97480705978e41fb1d9d8cd4094 WatchSource:0}: Error finding container b467fecfccc3e82d66a402895b2cac1734afa97480705978e41fb1d9d8cd4094: Status 404 returned error can't find the container with id b467fecfccc3e82d66a402895b2cac1734afa97480705978e41fb1d9d8cd4094
Jul 27 21:36:36.414231 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.414208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerStarted","Data":"b467fecfccc3e82d66a402895b2cac1734afa97480705978e41fb1d9d8cd4094"}
Jul 27 21:36:36.938128 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:36.938105    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:36:37.019788 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.019740    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config\") pod \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") "
Jul 27 21:36:37.019947 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.019801    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4rdn\" (UniqueName: \"kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn\") pod \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") "
Jul 27 21:36:37.019947 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.019846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") pod \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\" (UID: \"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b\") "
Jul 27 21:36:37.020134 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.020112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" (UID: "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:37.021721 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.021700    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn" (OuterVolumeSpecName: "kube-api-access-p4rdn") pod "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" (UID: "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b"). InnerVolumeSpecName "kube-api-access-p4rdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:37.021805 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.021745    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" (UID: "d4f550dd-99e5-4ddd-b84c-e89d1af94c2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:37.120660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.120638    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.120660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.120659    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rdn\" (UniqueName: \"kubernetes.io/projected/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-kube-api-access-p4rdn\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.120813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.120669    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.419560 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.419477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerStarted","Data":"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"}
Jul 27 21:36:37.419560 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.419545    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerStarted","Data":"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"}
Jul 27 21:36:37.420044 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.419583    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:37.420044 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.419616    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:37.421121 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421100    2555 generic.go:356] "Generic (PLEG): container finished" podID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerID="7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a" exitCode=0
Jul 27 21:36:37.421215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerDied","Data":"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"}
Jul 27 21:36:37.421215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7" event={"ID":"d4f550dd-99e5-4ddd-b84c-e89d1af94c2b","Type":"ContainerDied","Data":"9e8960133e9752c1969b4ff9dd4729ef759806d0c88a7fcf2d3f46fe8fc8ca81"}
Jul 27 21:36:37.421215 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421140    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:36:37.421322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421151    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"
Jul 27 21:36:37.421322 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.421234    2555 scope.go:117] "RemoveContainer" containerID="62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"
Jul 27 21:36:37.430258 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.430239    2555 scope.go:117] "RemoveContainer" containerID="7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"
Jul 27 21:36:37.437358 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.437315    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podStartSLOduration=3.437304138 podStartE2EDuration="3.437304138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:37.437281434 +0000 UTC m=+1961.107773099" watchObservedRunningTime="2026-07-27 21:36:37.437304138 +0000 UTC m=+1961.107795784"
Jul 27 21:36:37.437614 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.437593    2555 scope.go:117] "RemoveContainer" containerID="62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"
Jul 27 21:36:37.437924 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:37.437882    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0\": container with ID starting with 62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0 not found: ID does not exist" containerID="62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"
Jul 27 21:36:37.437924 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.437914    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0"} err="failed to get container status \"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0\": rpc error: code = NotFound desc = could not find container \"62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0\": container with ID starting with 62502e9214e66bd3b61a76d528b62b4b17b01d4c39ef469a689ef9a5269e8ee0 not found: ID does not exist"
Jul 27 21:36:37.438071 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.437938    2555 scope.go:117] "RemoveContainer" containerID="7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"
Jul 27 21:36:37.438200 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:36:37.438178    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a\": container with ID starting with 7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a not found: ID does not exist" containerID="7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"
Jul 27 21:36:37.438242 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.438209    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a"} err="failed to get container status \"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a\": rpc error: code = NotFound desc = could not find container \"7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a\": container with ID starting with 7f5ce82455d2977c85fbe9bce7e6372a28388bfd9b45dacc4e759a417f2d965a not found: ID does not exist"
Jul 27 21:36:37.453416 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.453393    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:36:37.456663 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:37.456643    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0f5d7-predictor-69c54f994c-99rb7"]
Jul 27 21:36:38.428676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:38.428627    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:36:38.858308 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:38.858273    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" path="/var/lib/kubelet/pods/d4f550dd-99e5-4ddd-b84c-e89d1af94c2b/volumes"
Jul 27 21:36:43.299584 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:43.299542    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 27 21:36:43.433318 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:43.433291    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:36:43.433783 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:43.433725    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:36:53.299970 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:53.299888    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:36:53.434321 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:36:53.434288    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:37:03.434199 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:03.434160    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:37:13.434488 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:13.434443    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 27 21:37:14.963931 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:14.963900    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:37:14.964335 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:14.964150    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container" containerID="cri-o://fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5" gracePeriod=30
Jul 27 21:37:14.964335 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:14.964197    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kube-rbac-proxy" containerID="cri-o://e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b" gracePeriod=30
Jul 27 21:37:15.030525 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030496    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:37:15.030881 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030863    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container"
Jul 27 21:37:15.030881 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030879    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container"
Jul 27 21:37:15.031027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030897    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kube-rbac-proxy"
Jul 27 21:37:15.031027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030902    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kube-rbac-proxy"
Jul 27 21:37:15.031027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030959    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kube-rbac-proxy"
Jul 27 21:37:15.031027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.030970    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f550dd-99e5-4ddd-b84c-e89d1af94c2b" containerName="kserve-container"
Jul 27 21:37:15.034346 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.034329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.037071 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.037032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2e96-predictor-serving-cert\""
Jul 27 21:37:15.037071 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.037051    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2e96-kube-rbac-proxy-sar-config\""
Jul 27 21:37:15.045995 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.045968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:37:15.096239 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.096213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.096351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.096251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.096351 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.096283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rcqj\" (UniqueName: \"kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.197594 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.197567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.197789 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.197618    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.197789 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.197668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rcqj\" (UniqueName: \"kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.197789 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:37:15.197717    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-serving-cert: secret "success-200-isvc-b2e96-predictor-serving-cert" not found
Jul 27 21:37:15.197993 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:37:15.197792    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls podName:77479cc9-99d5-49a3-b91a-ccdec2877ff3 nodeName:}" failed. No retries permitted until 2026-07-27 21:37:15.697770854 +0000 UTC m=+1999.368262479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls") pod "success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" (UID: "77479cc9-99d5-49a3-b91a-ccdec2877ff3") : secret "success-200-isvc-b2e96-predictor-serving-cert" not found
Jul 27 21:37:15.198230 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.198214    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.207035 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.207015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rcqj\" (UniqueName: \"kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.588710 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.588679    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce42dd03-8598-4512-9162-bc0232e54d77" containerID="e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b" exitCode=2
Jul 27 21:37:15.588879 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.588772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerDied","Data":"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"}
Jul 27 21:37:15.702041 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.702011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.704316 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.704297    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") pod \"success-200-isvc-b2e96-predictor-c4576f9f9-9xl48\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:15.946645 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:15.946579    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:16.070308 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.070285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:37:16.072226 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:37:16.072201    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77479cc9_99d5_49a3_b91a_ccdec2877ff3.slice/crio-6e675fb14d86128d6e31f3247f4800dc61397f364cefd562b04db7a6402d51f8 WatchSource:0}: Error finding container 6e675fb14d86128d6e31f3247f4800dc61397f364cefd562b04db7a6402d51f8: Status 404 returned error can't find the container with id 6e675fb14d86128d6e31f3247f4800dc61397f364cefd562b04db7a6402d51f8
Jul 27 21:37:16.595205 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.595171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerStarted","Data":"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"}
Jul 27 21:37:16.595390 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.595234    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerStarted","Data":"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"}
Jul 27 21:37:16.595390 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.595260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerStarted","Data":"6e675fb14d86128d6e31f3247f4800dc61397f364cefd562b04db7a6402d51f8"}
Jul 27 21:37:16.595390 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.595293    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:16.614376 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:16.614331    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podStartSLOduration=1.614314916 podStartE2EDuration="1.614314916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:16.613274494 +0000 UTC m=+2000.283766142" watchObservedRunningTime="2026-07-27 21:37:16.614314916 +0000 UTC m=+2000.284806562"
Jul 27 21:37:17.598920 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:17.598888    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:17.599913 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:17.599882    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:37:18.306718 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.306699    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:37:18.425818 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.425744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") pod \"ce42dd03-8598-4512-9162-bc0232e54d77\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") "
Jul 27 21:37:18.425917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.425839    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config\") pod \"ce42dd03-8598-4512-9162-bc0232e54d77\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") "
Jul 27 21:37:18.425917 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.425882    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg6mw\" (UniqueName: \"kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw\") pod \"ce42dd03-8598-4512-9162-bc0232e54d77\" (UID: \"ce42dd03-8598-4512-9162-bc0232e54d77\") "
Jul 27 21:37:18.426179 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.426158    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce42dd03-8598-4512-9162-bc0232e54d77" (UID: "ce42dd03-8598-4512-9162-bc0232e54d77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:18.427702 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.427671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce42dd03-8598-4512-9162-bc0232e54d77" (UID: "ce42dd03-8598-4512-9162-bc0232e54d77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:18.427813 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.427713    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw" (OuterVolumeSpecName: "kube-api-access-qg6mw") pod "ce42dd03-8598-4512-9162-bc0232e54d77" (UID: "ce42dd03-8598-4512-9162-bc0232e54d77"). InnerVolumeSpecName "kube-api-access-qg6mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:18.527129 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.527110    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42dd03-8598-4512-9162-bc0232e54d77-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.527230 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.527132    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42dd03-8598-4512-9162-bc0232e54d77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.527230 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.527142    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg6mw\" (UniqueName: \"kubernetes.io/projected/ce42dd03-8598-4512-9162-bc0232e54d77-kube-api-access-qg6mw\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.604330 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604305    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce42dd03-8598-4512-9162-bc0232e54d77" containerID="fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5" exitCode=0
Jul 27 21:37:18.604652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604349    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"
Jul 27 21:37:18.604652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604383    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerDied","Data":"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"}
Jul 27 21:37:18.604652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" event={"ID":"ce42dd03-8598-4512-9162-bc0232e54d77","Type":"ContainerDied","Data":"def6335986b00718b7cef81435677e99116bb3c2020e6ef53bf2a1a8c4e0ec9e"}
Jul 27 21:37:18.604652 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604465    2555 scope.go:117] "RemoveContainer" containerID="e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"
Jul 27 21:37:18.604966 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.604948    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:37:18.613126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.613109    2555 scope.go:117] "RemoveContainer" containerID="fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"
Jul 27 21:37:18.620136 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.620117    2555 scope.go:117] "RemoveContainer" containerID="e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"
Jul 27 21:37:18.620405 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:37:18.620383    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b\": container with ID starting with e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b not found: ID does not exist" containerID="e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"
Jul 27 21:37:18.620502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.620409    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b"} err="failed to get container status \"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b\": rpc error: code = NotFound desc = could not find container \"e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b\": container with ID starting with e18ba6ca10dcf6b07d27d285a5449e6a40b47cfa8c369cfa8db8e11db8b8bf1b not found: ID does not exist"
Jul 27 21:37:18.620502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.620425    2555 scope.go:117] "RemoveContainer" containerID="fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"
Jul 27 21:37:18.620669 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:37:18.620650    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5\": container with ID starting with fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5 not found: ID does not exist" containerID="fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"
Jul 27 21:37:18.620725 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.620678    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5"} err="failed to get container status \"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5\": rpc error: code = NotFound desc = could not find container \"fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5\": container with ID starting with fef2504242d172932e9f0660c0af191741228e6c2df1b52f70c4d711b512bce5 not found: ID does not exist"
Jul 27 21:37:18.625939 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.625919    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:37:18.631928 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.631898    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6"]
Jul 27 21:37:18.857980 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:18.857952    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" path="/var/lib/kubelet/pods/ce42dd03-8598-4512-9162-bc0232e54d77/volumes"
Jul 27 21:37:19.294886 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:19.294811    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61561-predictor-6959dd4cf7-jjqv6" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": context deadline exceeded"
Jul 27 21:37:23.434908 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:23.434874    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:37:23.608910 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:23.608876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:37:23.609386 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:23.609361    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:37:33.609788 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:33.609734    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:37:43.609582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:43.609539    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:37:53.609502 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:37:53.609462    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 27 21:38:03.610279 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:38:03.610244    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:46:29.805703 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:29.805668    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:46:29.808407 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:29.805922    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container" containerID="cri-o://3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59" gracePeriod=30
Jul 27 21:46:29.808407 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:29.805964    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kube-rbac-proxy" containerID="cri-o://333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2" gracePeriod=30
Jul 27 21:46:30.003057 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:30.003026    2555 generic.go:356] "Generic (PLEG): container finished" podID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerID="333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2" exitCode=2
Jul 27 21:46:30.003172 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:30.003063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerDied","Data":"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"}
Jul 27 21:46:32.345852 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.345830    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:46:32.420117 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.420096    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") pod \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") "
Jul 27 21:46:32.420224 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.420183    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rcqj\" (UniqueName: \"kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj\") pod \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") "
Jul 27 21:46:32.420224 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.420219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config\") pod \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\" (UID: \"77479cc9-99d5-49a3-b91a-ccdec2877ff3\") "
Jul 27 21:46:32.421620 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.420928    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77479cc9-99d5-49a3-b91a-ccdec2877ff3" (UID: "77479cc9-99d5-49a3-b91a-ccdec2877ff3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:46:32.424791 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.422695    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77479cc9-99d5-49a3-b91a-ccdec2877ff3" (UID: "77479cc9-99d5-49a3-b91a-ccdec2877ff3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:46:32.424791 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.422944    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj" (OuterVolumeSpecName: "kube-api-access-6rcqj") pod "77479cc9-99d5-49a3-b91a-ccdec2877ff3" (UID: "77479cc9-99d5-49a3-b91a-ccdec2877ff3"). InnerVolumeSpecName "kube-api-access-6rcqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:46:32.522067 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.522004    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77479cc9-99d5-49a3-b91a-ccdec2877ff3-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:46:32.522067 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.522037    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rcqj\" (UniqueName: \"kubernetes.io/projected/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-api-access-6rcqj\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:46:32.522067 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:32.522047    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77479cc9-99d5-49a3-b91a-ccdec2877ff3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:46:33.016875 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.016843    2555 generic.go:356] "Generic (PLEG): container finished" podID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerID="3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59" exitCode=0
Jul 27 21:46:33.016996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.016907    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"
Jul 27 21:46:33.016996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.016928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerDied","Data":"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"}
Jul 27 21:46:33.016996 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.016988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48" event={"ID":"77479cc9-99d5-49a3-b91a-ccdec2877ff3","Type":"ContainerDied","Data":"6e675fb14d86128d6e31f3247f4800dc61397f364cefd562b04db7a6402d51f8"}
Jul 27 21:46:33.017125 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.017018    2555 scope.go:117] "RemoveContainer" containerID="333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"
Jul 27 21:46:33.026660 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.026639    2555 scope.go:117] "RemoveContainer" containerID="3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"
Jul 27 21:46:33.034858 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.034840    2555 scope.go:117] "RemoveContainer" containerID="333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"
Jul 27 21:46:33.035128 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:46:33.035109    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2\": container with ID starting with 333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2 not found: ID does not exist" containerID="333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"
Jul 27 21:46:33.035175 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.035136    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2"} err="failed to get container status \"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2\": rpc error: code = NotFound desc = could not find container \"333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2\": container with ID starting with 333342385ab1a819669dde0417630fccdaa7a016038ccceabec844571c25cbf2 not found: ID does not exist"
Jul 27 21:46:33.035175 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.035152    2555 scope.go:117] "RemoveContainer" containerID="3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"
Jul 27 21:46:33.035416 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:46:33.035395    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59\": container with ID starting with 3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59 not found: ID does not exist" containerID="3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"
Jul 27 21:46:33.035471 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.035422    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59"} err="failed to get container status \"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59\": rpc error: code = NotFound desc = could not find container \"3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59\": container with ID starting with 3bf86a49c7096be18cb8888637156f59577f03bcbde7c598e57487ded7052f59 not found: ID does not exist"
Jul 27 21:46:33.035536 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.035515    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:46:33.038952 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:33.038930    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2e96-predictor-c4576f9f9-9xl48"]
Jul 27 21:46:34.857866 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:46:34.857826    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" path="/var/lib/kubelet/pods/77479cc9-99d5-49a3-b91a-ccdec2877ff3/volumes"
Jul 27 21:53:53.728024 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:53.727922    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:53:53.730621 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:53.728209    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container" containerID="cri-o://fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94" gracePeriod=30
Jul 27 21:53:53.730621 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:53.728272    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kube-rbac-proxy" containerID="cri-o://a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2" gracePeriod=30
Jul 27 21:53:53.874078 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:53.874046    2555 generic.go:356] "Generic (PLEG): container finished" podID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerID="a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2" exitCode=2
Jul 27 21:53:53.874222 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:53.874112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerDied","Data":"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"}
Jul 27 21:53:54.436017 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.435968    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b59xt/must-gather-ttjph"]
Jul 27 21:53:54.436320 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436309    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container"
Jul 27 21:53:54.436364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436321    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container"
Jul 27 21:53:54.436364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436333    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kube-rbac-proxy"
Jul 27 21:53:54.436364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436338    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kube-rbac-proxy"
Jul 27 21:53:54.436364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436349    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container"
Jul 27 21:53:54.436364 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436354    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436366    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kube-rbac-proxy"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436372    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kube-rbac-proxy"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436435    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kserve-container"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436444    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kserve-container"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436451    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce42dd03-8598-4512-9162-bc0232e54d77" containerName="kube-rbac-proxy"
Jul 27 21:53:54.436509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.436457    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="77479cc9-99d5-49a3-b91a-ccdec2877ff3" containerName="kube-rbac-proxy"
Jul 27 21:53:54.439565 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.439550    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.442449 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.442424    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b59xt\"/\"openshift-service-ca.crt\""
Jul 27 21:53:54.442582 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.442460    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b59xt\"/\"default-dockercfg-m99l2\""
Jul 27 21:53:54.443736 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.443718    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b59xt\"/\"kube-root-ca.crt\""
Jul 27 21:53:54.449413 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.449392    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b59xt/must-gather-ttjph"]
Jul 27 21:53:54.501786 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.501760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.501874 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.501859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.603003 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.602977    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.603090 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.603051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.603344 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.603319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.611407 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.611381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8\") pod \"must-gather-ttjph\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") " pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.762126 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.762072    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:53:54.889111 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.889087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b59xt/must-gather-ttjph"]
Jul 27 21:53:54.890712 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:53:54.890687    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14c6d9eb_1784_4dbe_b130_5fb40ac26e34.slice/crio-3642b94578ea1b77abd72a383a0cbe2c3a14591c74dd364bedfd0d3e986232db WatchSource:0}: Error finding container 3642b94578ea1b77abd72a383a0cbe2c3a14591c74dd364bedfd0d3e986232db: Status 404 returned error can't find the container with id 3642b94578ea1b77abd72a383a0cbe2c3a14591c74dd364bedfd0d3e986232db
Jul 27 21:53:54.892296 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:54.892278    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:53:55.890890 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:55.890742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b59xt/must-gather-ttjph" event={"ID":"14c6d9eb-1784-4dbe-b130-5fb40ac26e34","Type":"ContainerStarted","Data":"3642b94578ea1b77abd72a383a0cbe2c3a14591c74dd364bedfd0d3e986232db"}
Jul 27 21:53:56.415490 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.415465    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:53:56.521576 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.520675    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l246r\" (UniqueName: \"kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r\") pod \"a10c2012-9cac-49ee-9d83-fdccde059d06\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") "
Jul 27 21:53:56.521576 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.520809    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") pod \"a10c2012-9cac-49ee-9d83-fdccde059d06\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") "
Jul 27 21:53:56.521576 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.520852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config\") pod \"a10c2012-9cac-49ee-9d83-fdccde059d06\" (UID: \"a10c2012-9cac-49ee-9d83-fdccde059d06\") "
Jul 27 21:53:56.521576 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.521381    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a10c2012-9cac-49ee-9d83-fdccde059d06" (UID: "a10c2012-9cac-49ee-9d83-fdccde059d06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:53:56.526887 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.526845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r" (OuterVolumeSpecName: "kube-api-access-l246r") pod "a10c2012-9cac-49ee-9d83-fdccde059d06" (UID: "a10c2012-9cac-49ee-9d83-fdccde059d06"). InnerVolumeSpecName "kube-api-access-l246r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:53:56.528090 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.528055    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a10c2012-9cac-49ee-9d83-fdccde059d06" (UID: "a10c2012-9cac-49ee-9d83-fdccde059d06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:53:56.621959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.621897    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l246r\" (UniqueName: \"kubernetes.io/projected/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-api-access-l246r\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.621959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.621925    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10c2012-9cac-49ee-9d83-fdccde059d06-proxy-tls\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.621959 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.621939    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10c2012-9cac-49ee-9d83-fdccde059d06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.897319 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.897292    2555 generic.go:356] "Generic (PLEG): container finished" podID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerID="fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94" exitCode=0
Jul 27 21:53:56.897687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.897348    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"
Jul 27 21:53:56.897687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.897355    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerDied","Data":"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"}
Jul 27 21:53:56.897687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.897399    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm" event={"ID":"a10c2012-9cac-49ee-9d83-fdccde059d06","Type":"ContainerDied","Data":"b467fecfccc3e82d66a402895b2cac1734afa97480705978e41fb1d9d8cd4094"}
Jul 27 21:53:56.897687 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.897425    2555 scope.go:117] "RemoveContainer" containerID="a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"
Jul 27 21:53:56.906107 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.906090    2555 scope.go:117] "RemoveContainer" containerID="fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"
Jul 27 21:53:56.914566 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.914512    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:53:56.916105 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.916084    2555 scope.go:117] "RemoveContainer" containerID="a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"
Jul 27 21:53:56.916364 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:53:56.916342    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2\": container with ID starting with a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2 not found: ID does not exist" containerID="a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"
Jul 27 21:53:56.916452 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.916373    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2"} err="failed to get container status \"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2\": rpc error: code = NotFound desc = could not find container \"a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2\": container with ID starting with a2207d276a62667a6f08f528b1017c92f146e393f29daf81fc1b11596dd82ee2 not found: ID does not exist"
Jul 27 21:53:56.916452 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.916397    2555 scope.go:117] "RemoveContainer" containerID="fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"
Jul 27 21:53:56.916577 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.916556    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-beb6a-predictor-d9b4c445f-kbknm"]
Jul 27 21:53:56.916673 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:53:56.916629    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94\": container with ID starting with fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94 not found: ID does not exist" containerID="fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"
Jul 27 21:53:56.916673 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:56.916649    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94"} err="failed to get container status \"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94\": rpc error: code = NotFound desc = could not find container \"fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94\": container with ID starting with fea4164140d0dc224f899b229988734d3167e3657d3af504ddf4c853ea308a94 not found: ID does not exist"
Jul 27 21:53:58.858848 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:53:58.858817    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" path="/var/lib/kubelet/pods/a10c2012-9cac-49ee-9d83-fdccde059d06/volumes"
Jul 27 21:54:00.923422 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:00.923387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b59xt/must-gather-ttjph" event={"ID":"14c6d9eb-1784-4dbe-b130-5fb40ac26e34","Type":"ContainerStarted","Data":"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"}
Jul 27 21:54:01.929952 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:01.929919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b59xt/must-gather-ttjph" event={"ID":"14c6d9eb-1784-4dbe-b130-5fb40ac26e34","Type":"ContainerStarted","Data":"6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d"}
Jul 27 21:54:23.022814 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:23.022784    2555 generic.go:356] "Generic (PLEG): container finished" podID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerID="11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466" exitCode=0
Jul 27 21:54:23.023153 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:23.022821    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b59xt/must-gather-ttjph" event={"ID":"14c6d9eb-1784-4dbe-b130-5fb40ac26e34","Type":"ContainerDied","Data":"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"}
Jul 27 21:54:23.023153 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:23.023082    2555 scope.go:117] "RemoveContainer" containerID="11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"
Jul 27 21:54:23.907128 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:23.907097    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b59xt_must-gather-ttjph_14c6d9eb-1784-4dbe-b130-5fb40ac26e34/gather/0.log"
Jul 27 21:54:24.454372 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454342    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q2f4/must-gather-nth42"]
Jul 27 21:54:24.454730 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454715    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kube-rbac-proxy"
Jul 27 21:54:24.454818 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454731    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kube-rbac-proxy"
Jul 27 21:54:24.454818 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454788    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container"
Jul 27 21:54:24.454818 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454794    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container"
Jul 27 21:54:24.454978 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454893    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kube-rbac-proxy"
Jul 27 21:54:24.454978 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.454906    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10c2012-9cac-49ee-9d83-fdccde059d06" containerName="kserve-container"
Jul 27 21:54:24.457006 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.456990    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.459919 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.459896    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"openshift-service-ca.crt\""
Jul 27 21:54:24.460037 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.459996    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"kube-root-ca.crt\""
Jul 27 21:54:24.460037 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.460025    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q2f4\"/\"default-dockercfg-99gfb\""
Jul 27 21:54:24.466543 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.466524    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/must-gather-nth42"]
Jul 27 21:54:24.556185 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.556164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8wk\" (UniqueName: \"kubernetes.io/projected/304b58fd-fe71-41bc-aaf5-d1ea16272f86-kube-api-access-xd8wk\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.556291 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.556214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/304b58fd-fe71-41bc-aaf5-d1ea16272f86-must-gather-output\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.656904 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.656870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8wk\" (UniqueName: \"kubernetes.io/projected/304b58fd-fe71-41bc-aaf5-d1ea16272f86-kube-api-access-xd8wk\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.657012 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.656931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/304b58fd-fe71-41bc-aaf5-d1ea16272f86-must-gather-output\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.657197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.657180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/304b58fd-fe71-41bc-aaf5-d1ea16272f86-must-gather-output\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.665701 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.665678    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8wk\" (UniqueName: \"kubernetes.io/projected/304b58fd-fe71-41bc-aaf5-d1ea16272f86-kube-api-access-xd8wk\") pod \"must-gather-nth42\" (UID: \"304b58fd-fe71-41bc-aaf5-d1ea16272f86\") " pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.767193 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.767134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/must-gather-nth42"
Jul 27 21:54:24.901302 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:24.901225    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/must-gather-nth42"]
Jul 27 21:54:24.903417 ip-10-0-135-121 kubenswrapper[2555]: W0727 21:54:24.903393    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod304b58fd_fe71_41bc_aaf5_d1ea16272f86.slice/crio-079421a5ab75fb9abca6205a9deef7a26d460c2a3fd3df28b363911e504831c9 WatchSource:0}: Error finding container 079421a5ab75fb9abca6205a9deef7a26d460c2a3fd3df28b363911e504831c9: Status 404 returned error can't find the container with id 079421a5ab75fb9abca6205a9deef7a26d460c2a3fd3df28b363911e504831c9
Jul 27 21:54:25.033086 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:25.033030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/must-gather-nth42" event={"ID":"304b58fd-fe71-41bc-aaf5-d1ea16272f86","Type":"ContainerStarted","Data":"079421a5ab75fb9abca6205a9deef7a26d460c2a3fd3df28b363911e504831c9"}
Jul 27 21:54:26.040739 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:26.040165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/must-gather-nth42" event={"ID":"304b58fd-fe71-41bc-aaf5-d1ea16272f86","Type":"ContainerStarted","Data":"f5f3534c22a9834ba5a5bc2da6c32d113a237ac9d430317c6929c03bdbbf9d46"}
Jul 27 21:54:26.040739 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:26.040228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/must-gather-nth42" event={"ID":"304b58fd-fe71-41bc-aaf5-d1ea16272f86","Type":"ContainerStarted","Data":"59230a2580ba0d58ec82009fba101ecd81c8b86650496ae150ff124636e0ea9e"}
Jul 27 21:54:26.058509 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:26.058451    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q2f4/must-gather-nth42" podStartSLOduration=1.196955534 podStartE2EDuration="2.058432572s" totalImagesPullingTime="861.477038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:26.05640452 +0000 UTC m=+3029.726896168" watchObservedRunningTime="2026-07-27 21:54:26.058432572 +0000 UTC m=+3029.728924219"
Jul 27 21:54:27.161945 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:27.161911    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wffr5_639af4e4-909c-4f58-b7ca-f1c2ba762f03/global-pull-secret-syncer/0.log"
Jul 27 21:54:27.249476 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:27.249438    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jxdts_256d5c0d-52d6-4201-abc3-0354dada8960/konnectivity-agent/0.log"
Jul 27 21:54:27.341964 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:27.341939    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-121.ec2.internal_33a67532ab234867bd2a1f4dcc4b742b/haproxy/0.log"
Jul 27 21:54:29.303412 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.303378    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-b59xt/must-gather-ttjph"]
Jul 27 21:54:29.304369 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.304336    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-b59xt/must-gather-ttjph" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="copy" containerID="cri-o://6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d" gracePeriod=2
Jul 27 21:54:29.307585 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.307547    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-b59xt/must-gather-ttjph"]
Jul 27 21:54:29.309825 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:29.309572    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ttjph\" is forbidden: User \"system:node:ip-10-0-135-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b59xt\": no relationship found between node 'ip-10-0-135-121.ec2.internal' and this object" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:29.706773 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.705543    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b59xt_must-gather-ttjph_14c6d9eb-1784-4dbe-b130-5fb40ac26e34/copy/0.log"
Jul 27 21:54:29.710642 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.710381    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:29.715102 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:29.715057    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ttjph\" is forbidden: User \"system:node:ip-10-0-135-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b59xt\": no relationship found between node 'ip-10-0-135-121.ec2.internal' and this object" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:29.810897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.808967    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8\") pod \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") "
Jul 27 21:54:29.810897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.809031    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output\") pod \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\" (UID: \"14c6d9eb-1784-4dbe-b130-5fb40ac26e34\") "
Jul 27 21:54:29.810897 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.810563    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "14c6d9eb-1784-4dbe-b130-5fb40ac26e34" (UID: "14c6d9eb-1784-4dbe-b130-5fb40ac26e34"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:54:29.813598 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.813558    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8" (OuterVolumeSpecName: "kube-api-access-g5gp8") pod "14c6d9eb-1784-4dbe-b130-5fb40ac26e34" (UID: "14c6d9eb-1784-4dbe-b130-5fb40ac26e34"). InnerVolumeSpecName "kube-api-access-g5gp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:54:29.910715 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.910645    2555 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-must-gather-output\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:54:29.910715 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:29.910679    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/14c6d9eb-1784-4dbe-b130-5fb40ac26e34-kube-api-access-g5gp8\") on node \"ip-10-0-135-121.ec2.internal\" DevicePath \"\""
Jul 27 21:54:30.070326 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.066192    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b59xt_must-gather-ttjph_14c6d9eb-1784-4dbe-b130-5fb40ac26e34/copy/0.log"
Jul 27 21:54:30.070326 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.066609    2555 generic.go:356] "Generic (PLEG): container finished" podID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerID="6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d" exitCode=143
Jul 27 21:54:30.070326 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.066672    2555 scope.go:117] "RemoveContainer" containerID="6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d"
Jul 27 21:54:30.070326 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.066858    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:30.072941 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:30.072908    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ttjph\" is forbidden: User \"system:node:ip-10-0-135-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b59xt\": no relationship found between node 'ip-10-0-135-121.ec2.internal' and this object" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:30.087112 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:30.087074    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ttjph\" is forbidden: User \"system:node:ip-10-0-135-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b59xt\": no relationship found between node 'ip-10-0-135-121.ec2.internal' and this object" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" pod="openshift-must-gather-b59xt/must-gather-ttjph"
Jul 27 21:54:30.091057 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.090725    2555 scope.go:117] "RemoveContainer" containerID="11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"
Jul 27 21:54:30.111474 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.111370    2555 scope.go:117] "RemoveContainer" containerID="6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d"
Jul 27 21:54:30.111964 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:30.111933    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d\": container with ID starting with 6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d not found: ID does not exist" containerID="6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d"
Jul 27 21:54:30.112079 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.111979    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d"} err="failed to get container status \"6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d\": rpc error: code = NotFound desc = could not find container \"6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d\": container with ID starting with 6278e76369279821507801c00a828a7ffe74f93abc25a067c95d57ea2561170d not found: ID does not exist"
Jul 27 21:54:30.112079 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.112010    2555 scope.go:117] "RemoveContainer" containerID="11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"
Jul 27 21:54:30.112790 ip-10-0-135-121 kubenswrapper[2555]: E0727 21:54:30.112742    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466\": container with ID starting with 11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466 not found: ID does not exist" containerID="11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"
Jul 27 21:54:30.112889 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.112799    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466"} err="failed to get container status \"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466\": rpc error: code = NotFound desc = could not find container \"11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466\": container with ID starting with 11fda769ec1001ebf54c787675a4c22706d2acb9e1df66e2634ad18d485c3466 not found: ID does not exist"
Jul 27 21:54:30.862044 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:30.861963    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" path="/var/lib/kubelet/pods/14c6d9eb-1784-4dbe-b130-5fb40ac26e34/volumes"
Jul 27 21:54:31.317157 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.317121    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/alertmanager/0.log"
Jul 27 21:54:31.341214 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.341186    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/config-reloader/0.log"
Jul 27 21:54:31.365531 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.365446    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/kube-rbac-proxy-web/0.log"
Jul 27 21:54:31.391082 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.391008    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/kube-rbac-proxy/0.log"
Jul 27 21:54:31.440432 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.440338    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/kube-rbac-proxy-metric/0.log"
Jul 27 21:54:31.467053 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.467024    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/prom-label-proxy/0.log"
Jul 27 21:54:31.493789 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.493723    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1875ec7e-3d46-48b8-b007-a37707263403/init-config-reloader/0.log"
Jul 27 21:54:31.586187 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.586156    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fsfdn_4f0cb58b-ec9e-4825-a6d0-aae37b76e273/kube-state-metrics/0.log"
Jul 27 21:54:31.612098 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.612045    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fsfdn_4f0cb58b-ec9e-4825-a6d0-aae37b76e273/kube-rbac-proxy-main/0.log"
Jul 27 21:54:31.635457 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.635428    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fsfdn_4f0cb58b-ec9e-4825-a6d0-aae37b76e273/kube-rbac-proxy-self/0.log"
Jul 27 21:54:31.666668 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.666641    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-796887b949-2jbz9_b20c43c9-df65-4f26-b8b1-749689361936/metrics-server/0.log"
Jul 27 21:54:31.696814 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.696718    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-6mlss_138fc1a3-ebe3-45bc-a700-5f6f85a88410/monitoring-plugin/0.log"
Jul 27 21:54:31.890347 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.890316    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t9t54_565e4840-63af-4c39-94a1-677767f34256/node-exporter/0.log"
Jul 27 21:54:31.915766 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.915725    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t9t54_565e4840-63af-4c39-94a1-677767f34256/kube-rbac-proxy/0.log"
Jul 27 21:54:31.936158 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.936137    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t9t54_565e4840-63af-4c39-94a1-677767f34256/init-textfile/0.log"
Jul 27 21:54:31.962907 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.962824    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-77tmg_26c6dc33-1654-44e2-aa4e-3425a6003376/kube-rbac-proxy-main/0.log"
Jul 27 21:54:31.982027 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:31.981994    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-77tmg_26c6dc33-1654-44e2-aa4e-3425a6003376/kube-rbac-proxy-self/0.log"
Jul 27 21:54:32.007111 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.007065    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-77tmg_26c6dc33-1654-44e2-aa4e-3425a6003376/openshift-state-metrics/0.log"
Jul 27 21:54:32.049097 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.049063    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/prometheus/0.log"
Jul 27 21:54:32.073301 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.073271    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/config-reloader/0.log"
Jul 27 21:54:32.099392 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.099345    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/thanos-sidecar/0.log"
Jul 27 21:54:32.122268 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.122245    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/kube-rbac-proxy-web/0.log"
Jul 27 21:54:32.145716 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.145682    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/kube-rbac-proxy/0.log"
Jul 27 21:54:32.168679 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.168658    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/kube-rbac-proxy-thanos/0.log"
Jul 27 21:54:32.190641 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.190612    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e258328b-05ca-479a-bb31-35f38b079741/init-config-reloader/0.log"
Jul 27 21:54:32.226678 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.226573    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-srpsp_26fedb55-d188-40bf-a07d-9074c0a597db/prometheus-operator/0.log"
Jul 27 21:54:32.245110 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.245080    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-srpsp_26fedb55-d188-40bf-a07d-9074c0a597db/kube-rbac-proxy/0.log"
Jul 27 21:54:32.270358 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.270325    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-79w6s_e726fa5f-8c90-453a-95c4-6571d9ae2756/prometheus-operator-admission-webhook/0.log"
Jul 27 21:54:32.375130 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.375101    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/thanos-query/0.log"
Jul 27 21:54:32.398142 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.398118    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/kube-rbac-proxy-web/0.log"
Jul 27 21:54:32.420074 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.420043    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/kube-rbac-proxy/0.log"
Jul 27 21:54:32.443795 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.443735    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/prom-label-proxy/0.log"
Jul 27 21:54:32.476621 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.476590    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/kube-rbac-proxy-rules/0.log"
Jul 27 21:54:32.515423 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:32.515328    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f94c686b7-9fk92_7e60d566-2700-488e-aedd-56107c1a1c4a/kube-rbac-proxy-metrics/0.log"
Jul 27 21:54:34.965798 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.965740    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"]
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966104    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="gather"
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966115    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="gather"
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966134    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="copy"
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966139    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="copy"
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966191    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="copy"
Jul 27 21:54:34.966282 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.966199    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c6d9eb-1784-4dbe-b130-5fb40ac26e34" containerName="gather"
Jul 27 21:54:34.971644 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.971620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:34.975668 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:34.975641    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"]
Jul 27 21:54:35.060289 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.060251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-sys\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.060468 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.060308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ncd\" (UniqueName: \"kubernetes.io/projected/5644f6e1-5a98-447a-b114-4c21f9b32c44-kube-api-access-p5ncd\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.060468 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.060372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-proc\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.060468 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.060396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-lib-modules\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.060468 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.060449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-podres\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.161499 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.161459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-podres\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.161680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.161546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-sys\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.161680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.161585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ncd\" (UniqueName: \"kubernetes.io/projected/5644f6e1-5a98-447a-b114-4c21f9b32c44-kube-api-access-p5ncd\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.161680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.161636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-proc\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.161680 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.161663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-lib-modules\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.162090 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.162064    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-podres\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.162156 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.162141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-sys\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.162231 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.162206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-proc\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.162290 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.162223    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5644f6e1-5a98-447a-b114-4c21f9b32c44-lib-modules\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.171597 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.171567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ncd\" (UniqueName: \"kubernetes.io/projected/5644f6e1-5a98-447a-b114-4c21f9b32c44-kube-api-access-p5ncd\") pod \"perf-node-gather-daemonset-qvztk\" (UID: \"5644f6e1-5a98-447a-b114-4c21f9b32c44\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.286800 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.286713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:35.441635 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:35.441605    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"]
Jul 27 21:54:36.046831 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.046740    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6plj_b42f55a8-25f7-4a39-887a-aaf55094d967/dns/0.log"
Jul 27 21:54:36.064824 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.064802    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6plj_b42f55a8-25f7-4a39-887a-aaf55094d967/kube-rbac-proxy/0.log"
Jul 27 21:54:36.085251 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.085218    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f2l4_8699b98e-a1d2-4779-9534-29fddb721504/dns-node-resolver/0.log"
Jul 27 21:54:36.103157 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.103126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk" event={"ID":"5644f6e1-5a98-447a-b114-4c21f9b32c44","Type":"ContainerStarted","Data":"fbcb72d6a4486f81ada63b144f07d65ef0d9960c19d495029f54bec0bc76f88c"}
Jul 27 21:54:36.103324 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.103181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk" event={"ID":"5644f6e1-5a98-447a-b114-4c21f9b32c44","Type":"ContainerStarted","Data":"59c603306507a835bfa83dee8dcb1522e45af298aababac577a34ea7ca024d94"}
Jul 27 21:54:36.103324 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.103219    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:36.122313 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.122257    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk" podStartSLOduration=2.122239309 podStartE2EDuration="2.122239309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:36.119540749 +0000 UTC m=+3039.790032394" watchObservedRunningTime="2026-07-27 21:54:36.122239309 +0000 UTC m=+3039.792730955"
Jul 27 21:54:36.614676 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:36.614644    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6nvg_b32691ec-ae72-4d5b-a088-ed2b704d6b4d/node-ca/0.log"
Jul 27 21:54:37.681567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:37.681538    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g58pk_e842bd78-9c4e-47fe-b057-a3ab1db8d36b/serve-healthcheck-canary/0.log"
Jul 27 21:54:38.061869 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:38.061797    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6np85_77cabefb-b80f-42cf-8512-8635090b8873/kube-rbac-proxy/0.log"
Jul 27 21:54:38.080195 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:38.080171    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6np85_77cabefb-b80f-42cf-8512-8635090b8873/exporter/0.log"
Jul 27 21:54:38.100471 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:38.100448    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6np85_77cabefb-b80f-42cf-8512-8635090b8873/extractor/0.log"
Jul 27 21:54:40.338125 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:40.338102    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-wpn6v_b2745749-5692-43c4-add5-458b9adf7900/server/0.log"
Jul 27 21:54:40.507965 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:40.507935    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-2pqqc_826ae9c4-a8d8-4e3e-adb3-28203b7a9b54/manager/0.log"
Jul 27 21:54:40.547785 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:40.547759    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gmjzz_a60e6a8f-6118-43e6-ba87-80a85b512c1b/s3-init/0.log"
Jul 27 21:54:40.562200 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:40.562179    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gmjzz_a60e6a8f-6118-43e6-ba87-80a85b512c1b/download-hf-model/0.log"
Jul 27 21:54:40.590571 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:40.590513    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-7ghbb_f5cfa3e6-cf81-48c8-ab66-faa8fe982425/seaweedfs/0.log"
Jul 27 21:54:42.118191 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:42.118163    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qvztk"
Jul 27 21:54:45.465479 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.465454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:54:45.485077 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.485052    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/egress-router-binary-copy/0.log"
Jul 27 21:54:45.504299 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.504279    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/cni-plugins/0.log"
Jul 27 21:54:45.522841 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.522823    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/bond-cni-plugin/0.log"
Jul 27 21:54:45.543714 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.543694    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/routeoverride-cni/0.log"
Jul 27 21:54:45.563197 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.563173    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/whereabouts-cni-bincopy/0.log"
Jul 27 21:54:45.584480 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.584457    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2c5z6_c18b8d6e-6d55-4925-affb-96030d7814e3/whereabouts-cni/0.log"
Jul 27 21:54:45.924923 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:45.924891    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rwl5b_cd5121fa-06dc-40f8-844e-7b1237fbffa9/kube-multus/0.log"
Jul 27 21:54:46.003159 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.003128    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gb86p_585a203a-d420-4b43-b6f4-248884bede2b/network-metrics-daemon/0.log"
Jul 27 21:54:46.019920 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.019893    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gb86p_585a203a-d420-4b43-b6f4-248884bede2b/kube-rbac-proxy/0.log"
Jul 27 21:54:46.786911 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.786833    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/ovn-controller/0.log"
Jul 27 21:54:46.802306 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.802280    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/ovn-acl-logging/0.log"
Jul 27 21:54:46.823833 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.823807    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/kube-rbac-proxy-node/0.log"
Jul 27 21:54:46.843690 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.843670    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 21:54:46.863164 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.863146    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/northd/0.log"
Jul 27 21:54:46.882567 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.882547    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/nbdb/0.log"
Jul 27 21:54:46.903143 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.903125    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/sbdb/0.log"
Jul 27 21:54:46.963034 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:46.963012    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926vh_6cb44840-d1e7-4c34-8192-e2e93f2c1e55/ovnkube-controller/0.log"
Jul 27 21:54:48.349070 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:48.348996    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-d9xcx_2546be35-ea90-4047-8bdd-8c1f30697226/network-check-target-container/0.log"
Jul 27 21:54:49.216066 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:49.216033    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qk2hr_cff53865-e99c-4785-886a-13efb209469f/iptables-alerter/0.log"
Jul 27 21:54:49.819128 ip-10-0-135-121 kubenswrapper[2555]: I0727 21:54:49.819098    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7xlnp_da8288fe-6985-48bb-ad34-0cd66e4499b3/tuned/0.log"