Jul 07 02:15:54.671488 ip-10-0-135-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 02:15:54.671499 ip-10-0-135-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 02:15:54.671509 ip-10-0-135-57 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 02:15:54.671977 ip-10-0-135-57 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 02:16:04.888574 ip-10-0-135-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 02:16:04.888597 ip-10-0-135-57 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 09d2ad6b37fe4b7d91a0236812cb27dc --
Jul 07 02:18:07.536421 ip-10-0-135-57 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 02:18:07.981080 ip-10-0-135-57 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:07.981080 ip-10-0-135-57 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 02:18:07.981080 ip-10-0-135-57 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:07.981080 ip-10-0-135-57 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 02:18:07.981080 ip-10-0-135-57 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:07.982303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.982007    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 02:18:07.990279 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990247    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:07.990279 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990277    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:07.990279 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990281    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:07.990279 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990284    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990288    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990292    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990295    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990298    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990300    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990303    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990305    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990308    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990311    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990314    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990317    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990319    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990323    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990328    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990330    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990333    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990336    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990339    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990342    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:07.990431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990344    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990347    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990350    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990352    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990355    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990365    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990368    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990370    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990373    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990375    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990378    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990381    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990384    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990387    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990390    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990393    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990395    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990398    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990401    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990404    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:07.990914 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990406    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990412    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990415    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990417    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990422    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990424    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990427    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990429    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990432    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990435    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990438    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990440    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990443    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990446    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990448    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990452    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990454    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990457    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990459    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990462    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:07.991440 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990464    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990467    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990469    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990472    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990475    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990477    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990481    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990484    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990488    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990490    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990493    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990496    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990498    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990501    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990503    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990506    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990509    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990512    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990515    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:07.991939 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990517    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990521    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990525    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990528    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990531    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990534    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990537    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990539    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990542    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990544    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990547    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990549    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990552    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990554    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990556    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990559    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.990562    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991370    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991379    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:07.992518 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991385    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991390    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991394    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991397    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991400    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991403    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991406    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991409    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991412    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991414    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991417    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991420    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991423    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991426    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991429    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991431    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991434    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991437    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991439    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991442    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:07.992990 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991444    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991449    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991452    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991456    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991459    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991462    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991464    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991467    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991470    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991473    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991476    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991478    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991482    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991485    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991487    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991490    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991493    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991496    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991499    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:07.993512 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991502    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991505    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991507    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991510    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991512    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991515    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991518    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991521    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991524    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991526    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991529    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991531    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991534    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991537    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991540    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991543    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991545    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991547    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991550    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991552    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:07.994000 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991555    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991558    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991560    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991562    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991565    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991569    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991571    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991573    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991576    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991578    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991582    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991585    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991587    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991590    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991592    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991594    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991597    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991600    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991602    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:07.994520 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991605    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991608    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991610    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991613    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991615    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991620    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991623    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991626    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991629    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991632    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991634    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991640    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991643    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991645    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991648    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991650    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991653    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.991660    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:07.995008 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992088    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992093    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992097    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992100    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992102    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992105    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992108    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992110    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992112    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992115    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992118    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992120    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992122    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992126    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992128    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992131    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992134    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992136    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992139    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992141    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:07.995511 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992144    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992146    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992149    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992152    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992154    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992157    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992159    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992161    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992164    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992166    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992170    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992177    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992180    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992183    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992186    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992189    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992191    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992194    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992197    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992200    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:07.996001 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992203    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992206    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992209    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992211    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992214    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992216    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992219    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992221    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992224    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992226    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992229    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992231    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992233    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992236    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992238    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992240    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992243    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992246    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992248    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992252    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:07.996566 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992256    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992259    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992261    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992265    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992268    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992270    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992273    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992275    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992277    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992280    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992282    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992285    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992287    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992290    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992292    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992294    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992297    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992300    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992303    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992305    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:07.997066 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992308    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992310    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992313    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992315    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992318    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992320    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992323    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992327    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992331    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992334    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992337    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992339    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992342    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992344    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992347    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992350    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992353    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992355    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.992358    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992438    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 02:18:07.997598 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992456    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992462    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992467    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992471    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992474    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992479    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992483    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992486    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992489    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992493    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992496    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992499    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992503    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992506    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992509    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992511    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992514    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992520    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992523    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992528    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992531    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992535    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992539    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 02:18:07.998094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992542    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992545    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992549    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992552    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992555    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992558    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992561    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992564    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992569    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992572    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992575    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992577    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992581    2561 flags.go:64] FLAG: --enable-server="true"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992583    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992588    2561 flags.go:64] FLAG: --event-burst="100"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992590    2561 flags.go:64] FLAG: --event-qps="50"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992593    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992596    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992599    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992603    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992606    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992609    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992612    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992615    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992618    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 02:18:07.998654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992621    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992624    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992627    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992630    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992633    2561 flags.go:64] FLAG: --feature-gates=""
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992637    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992640    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992643    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992646    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992649    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992652    2561 flags.go:64] FLAG: --help="false"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992655    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-57.ec2.internal"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992659    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992665    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992668    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992672    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992675    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992678    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992681    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992684    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992686    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992689    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992693    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992695    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 07 02:18:07.999309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992698    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992701    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992704    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992707    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992710    2561 flags.go:64] FLAG: --lock-file=""
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992713    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992716    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992720    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992725    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992728    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992731    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992734    2561 flags.go:64] FLAG: --logging-format="text"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992737    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992741    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992743    2561 flags.go:64] FLAG: --manifest-url=""
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992746    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992750    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992753    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992757    2561 flags.go:64] FLAG: --max-pods="110"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992761    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992764    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992767    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992770    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992773    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992776    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 02:18:07.999974 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992779    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992786    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992789    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992792    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992795    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992798    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992803    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992806    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992809    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992812    2561 flags.go:64] FLAG: --port="10250"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992815    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992818    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d052533d5b29165e"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992821    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992824    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992827    2561 flags.go:64] FLAG: --register-node="true"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992831    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992835    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992837    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992840    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992843    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992847    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992850    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992853    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992856    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992858    2561 flags.go:64] FLAG: --runonce="false"
Jul 07 02:18:08.001119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992861    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992864    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992867    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992870    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992872    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992875    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992878    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992881    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992884    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992887    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992889    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992892    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992895    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992898    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992900    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992905    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992908    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992911    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992915    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992918    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992920    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992923    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992926    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992929    2561 flags.go:64] FLAG: --v="2"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992933    2561 flags.go:64] FLAG: --version="false"
Jul 07 02:18:08.002273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992937    2561 flags.go:64] FLAG: --vmodule=""
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992941    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.992944    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993076    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993081    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993085    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993088    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993091    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993093    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993097    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993100    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993103    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993105    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993108    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993110    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993113    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993115    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993118    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993120    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993123    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:08.003380 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993125    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993128    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993130    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993133    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993135    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993138    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993140    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993142    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993145    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993147    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993151    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993154    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993160    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993162    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993165    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993167    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993170    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993172    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993175    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:08.004079 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993177    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993180    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993184    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993186    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993190    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993194    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993197    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993200    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993203    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993206    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993209    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993212    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993214    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993217    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993220    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993223    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993226    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993229    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993231    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993234    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:08.004723 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993237    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993240    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993242    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993245    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993248    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993253    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993255    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993258    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993261    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993263    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993265    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993268    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993270    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993273    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993276    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993279    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993281    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993284    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993286    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993289    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:08.005644 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993291    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993293    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993296    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993298    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993301    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993303    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993306    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993308    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993311    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993314    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993316    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993319    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993321    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993324    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993327    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993329    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993332    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993337    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993339    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993342    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:08.006282 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993344    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993346    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:07.993350    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:07.994022    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.002196    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.002227    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.002256    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.002265    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002370    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002380    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002385    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002390    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002394    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002400    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:08.006912 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002404    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002408    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002412    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002417    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002421    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002425    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002429    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002433    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002437    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002441    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002445    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002449    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002454    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002458    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002462    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002467    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002471    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002475    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002479    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:08.007315 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002483    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002487    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002494    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002499    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002506    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002511    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002515    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002520    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002524    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002528    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002532    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002537    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002541    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002545    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002549    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002554    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002558    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002562    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002566    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002570    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:08.007791 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002574    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002578    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002583    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002587    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002591    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002595    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002599    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002603    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002607    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002614    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002618    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002622    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002626    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002630    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002634    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002638    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002643    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002650    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002657    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002661    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:08.008366 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002666    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002670    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002674    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002678    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002683    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002687    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002691    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002695    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002700    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002704    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002708    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002712    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002716    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002721    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002724    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002729    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002733    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002737    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002741    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002745    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:08.008858 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002749    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002753    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002758    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002762    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002766    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002770    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002774    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002778    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002783    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002787    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002791    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002795    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002800    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.002806    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:08.009426 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.002815    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003015    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003025    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003029    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003034    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003039    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003067    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003072    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003076    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003080    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003084    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003089    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003093    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003098    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003102    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003106    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003110    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003114    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003118    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003122    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:08.009815 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003126    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003131    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003135    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003140    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003144    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003148    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003153    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003157    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003161    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003165    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003169    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003173    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003178    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003182    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003186    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003190    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003194    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003199    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003203    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003207    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:08.010429 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003211    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003215    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003219    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003223    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003228    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003234    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003239    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003243    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003247    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003253    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003259    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003264    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003269    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003275    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003281    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003285    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003290    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003294    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003299    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:08.010919 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003303    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003308    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003312    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003316    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003320    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003324    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003328    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003332    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003336    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003339    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003343    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003348    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003352    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003356    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003360    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003364    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003368    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003372    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003376    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003380    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:08.011439 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003383    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003387    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003390    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003394    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003398    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003402    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003408    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003413    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003419    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003424    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003428    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003434    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003439    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003444    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003449    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003453    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003457    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003462    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003465    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:08.011955 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003470    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:08.003475    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.003482    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.003656    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.006582    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.008138    2561 server.go:1019] "Starting client certificate rotation"
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.008245    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:18:08.012443 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.008285    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:18:08.015192 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.015172    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:18:08.016358 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.016340    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:18:08.032452 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.032427    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 07 02:18:08.034679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.034664    2561 log.go:25] "Validated CRI v1 image API"
Jul 07 02:18:08.035493 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.035476    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 02:18:08.037985 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.037956    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a3864bc2-d6d8-4479-881f-32e5c6a096f6:/dev/nvme0n1p4 a8d358c9-8db4-4be1-9539-4d9ee7768a23:/dev/nvme0n1p3]
Jul 07 02:18:08.038081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.037986    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 02:18:08.043584 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.043566    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:18:08.043957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.043841    2561 manager.go:216] Machine: {Timestamp:2026-07-07 02:18:08.04234482 +0000 UTC m=+0.378239488 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100382 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e622af2dd3db4aec5f07a3ac04d4a SystemUUID:ec2e622a-f2dd-3db4-aec5-f07a3ac04d4a BootID:09d2ad6b-37fe-4b7d-91a0-236812cb27dc Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:e2:e5:07:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:e2:e5:07:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:5b:81:49:d1:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 02:18:08.043957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.043959    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 02:18:08.044089 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.044076    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 02:18:08.047510 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.047480    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 02:18:08.047653 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.047516    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 02:18:08.048215 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.048204    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 02:18:08.048255 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.048217    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 02:18:08.048255 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.048230    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:18:08.048255 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.048250    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:18:08.049679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.049661    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 02:18:08.053576 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053558    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:18:08.053707 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053697    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 02:18:08.053741 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053732    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 02:18:08.053769 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053753    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 02:18:08.053796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053774    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 02:18:08.053796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053787    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 07 02:18:08.053845 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.053822    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 02:18:08.054381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.054369    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:18:08.054433 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.054400    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:18:08.058282 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.058160    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 02:18:08.060190 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.060169    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 02:18:08.060277 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.060201    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 02:18:08.061550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061537    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 02:18:08.061587 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061560    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 02:18:08.061587 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061568    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 02:18:08.061587 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061574    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 02:18:08.061587 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061583    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061590    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061598    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061604    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061610    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061615    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061624    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 02:18:08.061696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.061629    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 02:18:08.062413 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.062402    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 02:18:08.062413 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.062412    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 02:18:08.065706 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.065691    2561 server.go:1268] "Started kubelet"
Jul 07 02:18:08.065817 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.065788    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 02:18:08.066633 ip-10-0-135-57 systemd[1]: Started Kubernetes Kubelet.
Jul 07 02:18:08.067368 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.067076    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 02:18:08.067368 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.067154    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 02:18:08.068114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.068098    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 02:18:08.068679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.068645    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 02:18:08.069376 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.069348    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 02:18:08.069376 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.069348    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 02:18:08.070170 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.070153    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 02:18:08.070276 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.070247    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 02:18:08.074739 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.073775    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-57.ec2.internal.18bfe061c9e4b005  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-57.ec2.internal,UID:ip-10-0-135-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-57.ec2.internal,},FirstTimestamp:2026-07-07 02:18:08.065662981 +0000 UTC m=+0.401557649,LastTimestamp:2026-07-07 02:18:08.065662981 +0000 UTC m=+0.401557649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-57.ec2.internal,}"
Jul 07 02:18:08.076132 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076112    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 02:18:08.076132 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076117    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 02:18:08.076296 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.076267    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.076835 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076817    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 02:18:08.076835 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076820    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 02:18:08.076954 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076851    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 02:18:08.077000 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.076990    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 02:18:08.077067 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077001    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 02:18:08.077173 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077155    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 02:18:08.077173 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077174    2561 factory.go:55] Registering systemd factory
Jul 07 02:18:08.077300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077185    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 07 02:18:08.077402 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077387    2561 factory.go:153] Registering CRI-O factory
Jul 07 02:18:08.077452 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077407    2561 factory.go:223] Registration of the crio container factory successfully
Jul 07 02:18:08.077452 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077443    2561 factory.go:103] Registering Raw factory
Jul 07 02:18:08.077529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.077459    2561 manager.go:1196] Started watching for new ooms in manager
Jul 07 02:18:08.078131 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.078120    2561 manager.go:349] Starting recovery of all containers
Jul 07 02:18:08.082287 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.082196    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 02:18:08.082474 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.082368    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 02:18:08.086213 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.086181    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cpkz5"
Jul 07 02:18:08.088152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.088133    2561 manager.go:354] Recovery completed
Jul 07 02:18:08.094142 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.094125    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.095666 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.095650    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cpkz5"
Jul 07 02:18:08.096260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096241    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.096310 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096278    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.096310 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096293    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.096821 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096805    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 02:18:08.096821 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096819    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 02:18:08.096923 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096826    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.096923 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.096837    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:18:08.097563 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097548    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.097619 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097577    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.097619 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097590    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.097921 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097910    2561 policy_none.go:49] "None policy: Start"
Jul 07 02:18:08.097959 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097927    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 02:18:08.097959 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.097938    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 02:18:08.098633 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.098564    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-57.ec2.internal.18bfe061cbb78f11  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-57.ec2.internal,UID:ip-10-0-135-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-57.ec2.internal,},FirstTimestamp:2026-07-07 02:18:08.096259857 +0000 UTC m=+0.432154534,LastTimestamp:2026-07-07 02:18:08.096259857 +0000 UTC m=+0.432154534,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-57.ec2.internal,}"
Jul 07 02:18:08.099122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.099110    2561 policy_none.go:47] "Start"
Jul 07 02:18:08.137834 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.137813    2561 manager.go:337] "Starting Device Plugin manager"
Jul 07 02:18:08.138009 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.137852    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 02:18:08.138009 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.137862    2561 server.go:87] "Starting device plugin registration server"
Jul 07 02:18:08.138193 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.138181    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 02:18:08.138246 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.138195    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 02:18:08.138438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.138415    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 02:18:08.138606 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.138516    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 02:18:08.138606 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.138524    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 02:18:08.138972 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.138953    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 02:18:08.139040 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.138998    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.238589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.238497    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.239992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.239524    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.239992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.239558    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.239992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.239570    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.239992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.239604    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.247867 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.247832    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 02:18:08.249257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.249230    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 02:18:08.249367 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.249278    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 02:18:08.249367 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.249307    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 02:18:08.249367 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.249351    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 02:18:08.251717 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.251698    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:08.253109 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.253092    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.253190 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.253117    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-57.ec2.internal\": node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.310826 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.310787    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.349904 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.349867    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"]
Jul 07 02:18:08.350093 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.349976    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.351728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.351711    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.351823 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.351747    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.351823 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.351761    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.353172 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353156    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.353300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.353355 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353304    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.353943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353922    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.354041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353950    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.354041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353959    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.354041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.353922    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.354041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.354027    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.354041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.354039    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.355487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.355471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.355537 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.355503    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:08.356192 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.356177    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:08.356282 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.356204    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:08.356282 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.356217    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:08.369691 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.369670    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-57.ec2.internal\" not found" node="ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.374272 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.374256    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-57.ec2.internal\" not found" node="ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.378444 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.378426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.378502 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.378452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.378502 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.378467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9df865c4a0e91b8563f787ba1a893c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-57.ec2.internal\" (UID: \"9df865c4a0e91b8563f787ba1a893c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.411697 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.411665    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.479100 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.479184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.479184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9df865c4a0e91b8563f787ba1a893c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-57.ec2.internal\" (UID: \"9df865c4a0e91b8563f787ba1a893c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.479184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479146    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9df865c4a0e91b8563f787ba1a893c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-57.ec2.internal\" (UID: \"9df865c4a0e91b8563f787ba1a893c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.479184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479168    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.479327 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:08.479171    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9889fb5d6f98348010daa3566e9aa02e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal\" (UID: \"9889fb5d6f98348010daa3566e9aa02e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:08.512204 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.512127    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:08.612975 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:08.612929    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:09.701727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:09.701697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:09.706916 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:09.706898    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:09.741670 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:09.741609    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:09.842225 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:09.842139    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:09.942745 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:09.942704    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:10.036216 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.036173    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 02:18:10.036755 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.036352    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 02:18:10.043342 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:10.043313    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:10.126992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.126921    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:13:09 +0000 UTC" deadline="2028-01-20 15:48:58.270731459 +0000 UTC"
Jul 07 02:18:10.126992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.126953    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13501h30m48.143781872s"
Jul 07 02:18:10.144101 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:10.144072    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:10.240896 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:10.240862    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9889fb5d6f98348010daa3566e9aa02e.slice/crio-a3a0717e2ec5a5fc47e3f668bc20999fe5c85723ead6d84ecf9644f5fd00ecf4 WatchSource:0}: Error finding container a3a0717e2ec5a5fc47e3f668bc20999fe5c85723ead6d84ecf9644f5fd00ecf4: Status 404 returned error can't find the container with id a3a0717e2ec5a5fc47e3f668bc20999fe5c85723ead6d84ecf9644f5fd00ecf4
Jul 07 02:18:10.241431 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:10.241411    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df865c4a0e91b8563f787ba1a893c2f.slice/crio-05eb972768ca21a06d730ced5a7906ffd44251619701ce3a30a0fc5be0ae11af WatchSource:0}: Error finding container 05eb972768ca21a06d730ced5a7906ffd44251619701ce3a30a0fc5be0ae11af: Status 404 returned error can't find the container with id 05eb972768ca21a06d730ced5a7906ffd44251619701ce3a30a0fc5be0ae11af
Jul 07 02:18:10.244214 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:10.244194    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-57.ec2.internal\" not found"
Jul 07 02:18:10.245410 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.245395    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:18:10.281936 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.281878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal" event={"ID":"9889fb5d6f98348010daa3566e9aa02e","Type":"ContainerStarted","Data":"a3a0717e2ec5a5fc47e3f668bc20999fe5c85723ead6d84ecf9644f5fd00ecf4"}
Jul 07 02:18:10.283400 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.283376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal" event={"ID":"9df865c4a0e91b8563f787ba1a893c2f","Type":"ContainerStarted","Data":"05eb972768ca21a06d730ced5a7906ffd44251619701ce3a30a0fc5be0ae11af"}
Jul 07 02:18:10.324215 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.324188    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:10.405403 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.405306    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:10.418951 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.418930    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:18:10.420830 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.420816    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal"
Jul 07 02:18:10.429654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.429635    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:18:10.472887 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.472858    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:10.607543 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.607512    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:10.847088 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:10.846993    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:11.082892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.082858    2561 apiserver.go:52] "Watching apiserver"
Jul 07 02:18:11.091058 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.091026    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 02:18:11.093408 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.093376    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gqwgh","openshift-multus/network-metrics-daemon-t5bd7","openshift-network-diagnostics/network-check-target-r2x82","openshift-network-operator/iptables-alerter-wdnwc","openshift-ovn-kubernetes/ovnkube-node-9q4km","kube-system/global-pull-secret-syncer-dqm7c","kube-system/konnectivity-agent-q8szf","kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q","openshift-dns/node-resolver-t2mlv","openshift-image-registry/node-ca-ngcnf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal","openshift-multus/multus-additional-cni-plugins-6jngg","openshift-cluster-node-tuning-operator/tuned-4h5hb"]
Jul 07 02:18:11.095257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.095230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.097154 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.097108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.097933 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.097915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 02:18:11.098033 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.097957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5n9w\""
Jul 07 02:18:11.098092 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.098083    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 02:18:11.098577 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.098556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:11.098737 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.098652    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:11.099461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.099442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.099749 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.099728    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 02:18:11.100046 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.100027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tqzxv\""
Jul 07 02:18:11.100418 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.100391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 02:18:11.100508 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.100425    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 02:18:11.100508 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.100427    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.101447 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.101424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.103081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.103065    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.103892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.103794    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 02:18:11.103892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.103865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.105011 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.104007    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.105011 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.104008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wtl6t\""
Jul 07 02:18:11.105011 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.104579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.105011 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.104653    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:11.105230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.105184    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 02:18:11.105677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.105458    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 02:18:11.106055 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.106037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.106136 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.106056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.106302 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.106287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.107231 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.107214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 02:18:11.107314 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.107278    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 02:18:11.107314 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.107308    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-45rcd\""
Jul 07 02:18:11.107537 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.107515    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 02:18:11.107932 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.107917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.108928 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.108907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.109135 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.109116    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 02:18:11.109223 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.109120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.109525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.109504    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-68wjn\""
Jul 07 02:18:11.109812 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.109795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.109893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.109861    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:11.110530 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.110462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7l7f9\""
Jul 07 02:18:11.110798 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.110763    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 02:18:11.111322 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.111270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.113051 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.112715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.114462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.114442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.114612 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.114445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.114970 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.114935    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7l6j8\""
Jul 07 02:18:11.115249 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.115233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4m52t\""
Jul 07 02:18:11.115249 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.115255    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.115393 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.115238    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 02:18:11.115796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.115651    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.115796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.115732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.116700 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-systemd-units\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.116803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-etc-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.116803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-env-overrides\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.116803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-system-cni-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.116803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea23be-df66-49b6-8ebc-efb851af5659-iptables-alerter-script\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-registration-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-sys-fs\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96n9\" (UniqueName: \"kubernetes.io/projected/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kube-api-access-g96n9\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-kubelet\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-log-socket\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.116978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-bin\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea23be-df66-49b6-8ebc-efb851af5659-host-slash\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.117394 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.117394 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-systemd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117681 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-netd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117681 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117681 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-node-log\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117681 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-netns\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-config\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnbm\" (UniqueName: \"kubernetes.io/projected/4756d2f8-25d8-497e-be3e-d934a593ccf4-kube-api-access-pvnbm\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vs2x\" (UniqueName: \"kubernetes.io/projected/67ea23be-df66-49b6-8ebc-efb851af5659-kube-api-access-7vs2x\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-dbus\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.117900 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-slash\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-ovn\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovn-node-metrics-cert\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.117989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118073    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e00c637-d42b-4518-bd76-555484211f93-konnectivity-ca\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-script-lib\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-cnibin\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-os-release\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.118228 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:11.118597 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.118706 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9c5\" (UniqueName: \"kubernetes.io/projected/d3172c72-6087-4af8-bf4f-24475348408a-kube-api-access-nc9c5\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.118856 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.118930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-device-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-kubelet-config\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.118991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.119010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-socket-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.119026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e00c637-d42b-4518-bd76-555484211f93-agent-certs\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.119045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-var-lib-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.119049 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.119048    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:11.119603 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.119587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8mlzf\""
Jul 07 02:18:11.126061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.125911    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:18:11.146462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.146416    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-khkgz"
Jul 07 02:18:11.155262 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.155239    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-khkgz"
Jul 07 02:18:11.206410 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.206382    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 02:18:11.219351 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-cnibin\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.219351 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9c5\" (UniqueName: \"kubernetes.io/projected/d3172c72-6087-4af8-bf4f-24475348408a-kube-api-access-nc9c5\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.219351 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-cnibin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.219698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-modprobe-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.219698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-env-overrides\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.219698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.219698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219496    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-cnibin\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.219698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-system-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-device-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-kubelet-config\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-systemd-units\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-system-cni-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-socket-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.219938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e00c637-d42b-4518-bd76-555484211f93-agent-certs\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.219960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-var-lib-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-multus\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-hostroot\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220035    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-env-overrides\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220048    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff16fd4f-d7f3-4591-a4fe-8b8905692255-tmp-dir\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220133    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-device-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220151    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-systemd-units\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220187    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-system-cni-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-registration-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220237    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-socket-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-netd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220288    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-kubelet-config\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220338    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-var-lib-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220347    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-registration-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220385    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.220409    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.220480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220410    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-etc-kubernetes\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhcv\" (UniqueName: \"kubernetes.io/projected/49f46c00-1bba-4491-aa68-aef5b6d3c490-kube-api-access-8lhcv\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.220489    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.720472287 +0000 UTC m=+3.027790697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220519    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-netd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-run\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-var-lib-kubelet\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea23be-df66-49b6-8ebc-efb851af5659-host-slash\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-systemd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-netns\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-conf-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-systemd\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220794    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea23be-df66-49b6-8ebc-efb851af5659-host-slash\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d816096e-154e-490a-9c94-815e35733b2f-serviceca\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-kubernetes\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.221311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-tmp\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-netns\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-cni-binary-copy\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-slash\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220971    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-ovn\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221012    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-run-netns\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221044    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-slash\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.220925    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-socket-dir-parent\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221076    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-run-ovn\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-k8s-cni-cncf-io\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c66\" (UniqueName: \"kubernetes.io/projected/8f161180-18e0-47b7-af90-2603fa058c4f-kube-api-access-l5c66\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.222119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d816096e-154e-490a-9c94-815e35733b2f-host\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-systemd\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-os-release\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-kubelet\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-daemon-config\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4lj\" (UniqueName: \"kubernetes.io/projected/d816096e-154e-490a-9c94-815e35733b2f-kube-api-access-cw4lj\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221292    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221390    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-os-release\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221444    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3172c72-6087-4af8-bf4f-24475348408a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-lib-modules\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-etc-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-host\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221573    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-etc-openvswitch\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea23be-df66-49b6-8ebc-efb851af5659-iptables-alerter-script\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.222578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221530    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3172c72-6087-4af8-bf4f-24475348408a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-sys-fs\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96n9\" (UniqueName: \"kubernetes.io/projected/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kube-api-access-g96n9\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-kubelet\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221708    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-sys-fs\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-log-socket\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-bin\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221748    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-log-socket\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-bin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221831    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-kubelet\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-multus-certs\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-sys\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-etc-tuned\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-node-log\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-config\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221982    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnbm\" (UniqueName: \"kubernetes.io/projected/4756d2f8-25d8-497e-be3e-d934a593ccf4-kube-api-access-pvnbm\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysconfig\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222089    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-node-log\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vs2x\" (UniqueName: \"kubernetes.io/projected/67ea23be-df66-49b6-8ebc-efb851af5659-kube-api-access-7vs2x\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.221874    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-cni-bin\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222134    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4756d2f8-25d8-497e-be3e-d934a593ccf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-dbus\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovn-node-metrics-cert\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-os-release\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222272    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea23be-df66-49b6-8ebc-efb851af5659-iptables-alerter-script\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq64r\" (UniqueName: \"kubernetes.io/projected/ed4f47c0-1a2e-4045-826a-65ba949859d3-kube-api-access-wq64r\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222342    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb558531-20cf-4b45-a955-5f1ca96e1516-dbus\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff16fd4f-d7f3-4591-a4fe-8b8905692255-hosts-file\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e00c637-d42b-4518-bd76-555484211f93-konnectivity-ca\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-script-lib\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.223697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dc2p\" (UniqueName: \"kubernetes.io/projected/ff16fd4f-d7f3-4591-a4fe-8b8905692255-kube-api-access-4dc2p\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.224292 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-conf\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.224292 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222508    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-config\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.224292 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222945    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e00c637-d42b-4518-bd76-555484211f93-konnectivity-ca\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.224292 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.222970    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovnkube-script-lib\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.225599 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.225574    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4756d2f8-25d8-497e-be3e-d934a593ccf4-ovn-node-metrics-cert\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.227549 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.227525    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:11.227673 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.227557    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:11.227673 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.227572    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.227673 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.227662    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.727645172 +0000 UTC m=+3.034963556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.227841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.227826    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e00c637-d42b-4518-bd76-555484211f93-agent-certs\") pod \"konnectivity-agent-q8szf\" (UID: \"0e00c637-d42b-4518-bd76-555484211f93\") " pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.228311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.228257    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9c5\" (UniqueName: \"kubernetes.io/projected/d3172c72-6087-4af8-bf4f-24475348408a-kube-api-access-nc9c5\") pod \"multus-additional-cni-plugins-6jngg\" (UID: \"d3172c72-6087-4af8-bf4f-24475348408a\") " pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.231268 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.231230    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96n9\" (UniqueName: \"kubernetes.io/projected/b1c05da0-30a5-4cd3-a6f4-1e4c5201d117-kube-api-access-g96n9\") pod \"aws-ebs-csi-driver-node-6vj9q\" (UID: \"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.231810 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.231789    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vs2x\" (UniqueName: \"kubernetes.io/projected/67ea23be-df66-49b6-8ebc-efb851af5659-kube-api-access-7vs2x\") pod \"iptables-alerter-wdnwc\" (UID: \"67ea23be-df66-49b6-8ebc-efb851af5659\") " pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.234136 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.234115    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnbm\" (UniqueName: \"kubernetes.io/projected/4756d2f8-25d8-497e-be3e-d934a593ccf4-kube-api-access-pvnbm\") pod \"ovnkube-node-9q4km\" (UID: \"4756d2f8-25d8-497e-be3e-d934a593ccf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.323722 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-cnibin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.323722 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-modprobe-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-system-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323843    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-cnibin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323895    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-modprobe-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-multus\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-hostroot\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.323961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323953    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-system-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff16fd4f-d7f3-4591-a4fe-8b8905692255-tmp-dir\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-etc-kubernetes\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.323997    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-multus\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhcv\" (UniqueName: \"kubernetes.io/projected/49f46c00-1bba-4491-aa68-aef5b6d3c490-kube-api-access-8lhcv\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324052    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-hostroot\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324066    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-etc-kubernetes\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-run\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-var-lib-kubelet\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324184    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-d\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-netns\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-conf-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324236    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-run\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324241    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-var-lib-kubelet\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324277    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-netns\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d816096e-154e-490a-9c94-815e35733b2f-serviceca\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324286    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-conf-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-kubernetes\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-tmp\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-cni-binary-copy\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-socket-dir-parent\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-k8s-cni-cncf-io\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324416    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff16fd4f-d7f3-4591-a4fe-8b8905692255-tmp-dir\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324502    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-kubernetes\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324554    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-socket-dir-parent\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c66\" (UniqueName: \"kubernetes.io/projected/8f161180-18e0-47b7-af90-2603fa058c4f-kube-api-access-l5c66\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324601    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d816096e-154e-490a-9c94-815e35733b2f-serviceca\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d816096e-154e-490a-9c94-815e35733b2f-host\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-systemd\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.324872 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-kubelet\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-daemon-config\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4lj\" (UniqueName: \"kubernetes.io/projected/d816096e-154e-490a-9c94-815e35733b2f-kube-api-access-cw4lj\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-lib-modules\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-host\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.324850    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-bin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324886    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d816096e-154e-490a-9c94-815e35733b2f-host\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-multus-certs\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.324914    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.824895276 +0000 UTC m=+3.132213661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-cni-binary-copy\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324928    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-multus-certs\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324957    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-systemd\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.324977    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-host\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325000    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-cni-dir\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325017    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-lib-modules\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325023    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-cni-bin\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.325732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325067    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-var-lib-kubelet\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-sys\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-etc-tuned\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325176    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-sys\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysconfig\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-os-release\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq64r\" (UniqueName: \"kubernetes.io/projected/ed4f47c0-1a2e-4045-826a-65ba949859d3-kube-api-access-wq64r\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff16fd4f-d7f3-4591-a4fe-8b8905692255-hosts-file\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325268    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysconfig\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dc2p\" (UniqueName: \"kubernetes.io/projected/ff16fd4f-d7f3-4591-a4fe-8b8905692255-kube-api-access-4dc2p\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325309    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-os-release\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-conf\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325338    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff16fd4f-d7f3-4591-a4fe-8b8905692255-hosts-file\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325462    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49f46c00-1bba-4491-aa68-aef5b6d3c490-multus-daemon-config\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325471    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f161180-18e0-47b7-af90-2603fa058c4f-etc-sysctl-conf\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.326260 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.325588    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49f46c00-1bba-4491-aa68-aef5b6d3c490-host-run-k8s-cni-cncf-io\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.327285 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.327256    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-tmp\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.327658 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.327636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f161180-18e0-47b7-af90-2603fa058c4f-etc-tuned\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.336585 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.336538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c66\" (UniqueName: \"kubernetes.io/projected/8f161180-18e0-47b7-af90-2603fa058c4f-kube-api-access-l5c66\") pod \"tuned-4h5hb\" (UID: \"8f161180-18e0-47b7-af90-2603fa058c4f\") " pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.336976 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.336955    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dc2p\" (UniqueName: \"kubernetes.io/projected/ff16fd4f-d7f3-4591-a4fe-8b8905692255-kube-api-access-4dc2p\") pod \"node-resolver-t2mlv\" (UID: \"ff16fd4f-d7f3-4591-a4fe-8b8905692255\") " pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.338188 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.338139    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq64r\" (UniqueName: \"kubernetes.io/projected/ed4f47c0-1a2e-4045-826a-65ba949859d3-kube-api-access-wq64r\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.338438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.338413    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4lj\" (UniqueName: \"kubernetes.io/projected/d816096e-154e-490a-9c94-815e35733b2f-kube-api-access-cw4lj\") pod \"node-ca-ngcnf\" (UID: \"d816096e-154e-490a-9c94-815e35733b2f\") " pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.339385 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.339363    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhcv\" (UniqueName: \"kubernetes.io/projected/49f46c00-1bba-4491-aa68-aef5b6d3c490-kube-api-access-8lhcv\") pod \"multus-gqwgh\" (UID: \"49f46c00-1bba-4491-aa68-aef5b6d3c490\") " pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.407579 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.407496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:11.416636 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.416597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jngg"
Jul 07 02:18:11.433298 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.433264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdnwc"
Jul 07 02:18:11.439360 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.439338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:11.447047 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.447018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q"
Jul 07 02:18:11.454317 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.454290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqwgh"
Jul 07 02:18:11.465796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.465767    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2mlv"
Jul 07 02:18:11.471846 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.471821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ngcnf"
Jul 07 02:18:11.477728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.477701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4h5hb"
Jul 07 02:18:11.728659 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.728543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:11.728659 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.728654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728737    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728763    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728761    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728772    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728832    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:12.728813205 +0000 UTC m=+4.036131579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.728893 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.728852    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:12.728842368 +0000 UTC m=+4.036160740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.829732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:11.829444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:11.829732 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.829548    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:11.829732 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:11.829596    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:12.82958188 +0000 UTC m=+4.136900248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:12.157840 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.157752    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:13:11 +0000 UTC" deadline="2028-03-24 16:39:04.68655462 +0000 UTC"
Jul 07 02:18:12.157840 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.157787    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15038h20m52.528771473s"
Jul 07 02:18:12.278163 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.278135    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:12.278325 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.278139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:12.278325 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.278250    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:12.278442 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.278330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:12.325348 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.325310    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3172c72_6087_4af8_bf4f_24475348408a.slice/crio-950b3c5940e91314a13182592de79fcdf5c70b8bad44dbab9528270226f885a4 WatchSource:0}: Error finding container 950b3c5940e91314a13182592de79fcdf5c70b8bad44dbab9528270226f885a4: Status 404 returned error can't find the container with id 950b3c5940e91314a13182592de79fcdf5c70b8bad44dbab9528270226f885a4
Jul 07 02:18:12.326027 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.325998    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f46c00_1bba_4491_aa68_aef5b6d3c490.slice/crio-0bae2e66717aaa46d5c4c1fb5e741533252a5f4402b1fe90f5fb3392f4b6ad51 WatchSource:0}: Error finding container 0bae2e66717aaa46d5c4c1fb5e741533252a5f4402b1fe90f5fb3392f4b6ad51: Status 404 returned error can't find the container with id 0bae2e66717aaa46d5c4c1fb5e741533252a5f4402b1fe90f5fb3392f4b6ad51
Jul 07 02:18:12.329265 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.329224    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c05da0_30a5_4cd3_a6f4_1e4c5201d117.slice/crio-6a5f91e06f324953479ba66d62450b6e2a1fa7aa86f2ac92823d6c828a5c0e36 WatchSource:0}: Error finding container 6a5f91e06f324953479ba66d62450b6e2a1fa7aa86f2ac92823d6c828a5c0e36: Status 404 returned error can't find the container with id 6a5f91e06f324953479ba66d62450b6e2a1fa7aa86f2ac92823d6c828a5c0e36
Jul 07 02:18:12.337977 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.337957    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e00c637_d42b_4518_bd76_555484211f93.slice/crio-3b70e33202e672e08097f880c504011c43d84d98cd56360b1c2a4362545dce2f WatchSource:0}: Error finding container 3b70e33202e672e08097f880c504011c43d84d98cd56360b1c2a4362545dce2f: Status 404 returned error can't find the container with id 3b70e33202e672e08097f880c504011c43d84d98cd56360b1c2a4362545dce2f
Jul 07 02:18:12.340189 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.339601    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f161180_18e0_47b7_af90_2603fa058c4f.slice/crio-842c630b9c627dd18a5ae14050b8574309b6dfdc166fdbcaf5917703c60b02cd WatchSource:0}: Error finding container 842c630b9c627dd18a5ae14050b8574309b6dfdc166fdbcaf5917703c60b02cd: Status 404 returned error can't find the container with id 842c630b9c627dd18a5ae14050b8574309b6dfdc166fdbcaf5917703c60b02cd
Jul 07 02:18:12.346260 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:18:12.346203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4756d2f8_25d8_497e_be3e_d934a593ccf4.slice/crio-7182459e953b99f7b0500fa3b141bba33cbe78f0daf436c6f7fb250937afe4d9 WatchSource:0}: Error finding container 7182459e953b99f7b0500fa3b141bba33cbe78f0daf436c6f7fb250937afe4d9: Status 404 returned error can't find the container with id 7182459e953b99f7b0500fa3b141bba33cbe78f0daf436c6f7fb250937afe4d9
Jul 07 02:18:12.738197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.737897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:12.738355 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.737957    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:12.738355 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.738284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:12.738355 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.738317    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:12.738355 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.738335    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:12.738509 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.738391    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:14.738377336 +0000 UTC m=+6.045695704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:12.738509 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.738428    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:12.738509 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.738500    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:14.738468607 +0000 UTC m=+6.045786980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:12.839672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:12.838960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:12.839672 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.839171    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:12.839672 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:12.839237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:14.839218551 +0000 UTC m=+6.146536920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:13.158956 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.158660    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:13:11 +0000 UTC" deadline="2028-03-26 03:48:03.626180014 +0000 UTC"
Jul 07 02:18:13.158956 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.158705    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15073h29m50.467481725s"
Jul 07 02:18:13.283110 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.283064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:13.283419 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:13.283390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:13.306398 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.305359    2561 generic.go:356] "Generic (PLEG): container finished" podID="9889fb5d6f98348010daa3566e9aa02e" containerID="97afa9068c8e87f78ad6fa199680e7413cc7525d7f6093476a7c3e2daa65936f" exitCode=0
Jul 07 02:18:13.306398 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.305466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal" event={"ID":"9889fb5d6f98348010daa3566e9aa02e","Type":"ContainerDied","Data":"97afa9068c8e87f78ad6fa199680e7413cc7525d7f6093476a7c3e2daa65936f"}
Jul 07 02:18:13.328249 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.328160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4h5hb" event={"ID":"8f161180-18e0-47b7-af90-2603fa058c4f","Type":"ContainerStarted","Data":"842c630b9c627dd18a5ae14050b8574309b6dfdc166fdbcaf5917703c60b02cd"}
Jul 07 02:18:13.338334 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.338297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"7182459e953b99f7b0500fa3b141bba33cbe78f0daf436c6f7fb250937afe4d9"}
Jul 07 02:18:13.355300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.355258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdnwc" event={"ID":"67ea23be-df66-49b6-8ebc-efb851af5659","Type":"ContainerStarted","Data":"73468075fbba5e0b5f71d2aca039e7f03f9a76096376721ac08c8b72af6b989f"}
Jul 07 02:18:13.368367 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.368320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ngcnf" event={"ID":"d816096e-154e-490a-9c94-815e35733b2f","Type":"ContainerStarted","Data":"a98f09348d7469aecd6dbcd5ecb048c243968f061d7653bd25c3b833905eec94"}
Jul 07 02:18:13.371242 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.371204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8szf" event={"ID":"0e00c637-d42b-4518-bd76-555484211f93","Type":"ContainerStarted","Data":"3b70e33202e672e08097f880c504011c43d84d98cd56360b1c2a4362545dce2f"}
Jul 07 02:18:13.385332 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.385196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q" event={"ID":"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117","Type":"ContainerStarted","Data":"6a5f91e06f324953479ba66d62450b6e2a1fa7aa86f2ac92823d6c828a5c0e36"}
Jul 07 02:18:13.399562 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.399518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqwgh" event={"ID":"49f46c00-1bba-4491-aa68-aef5b6d3c490","Type":"ContainerStarted","Data":"0bae2e66717aaa46d5c4c1fb5e741533252a5f4402b1fe90f5fb3392f4b6ad51"}
Jul 07 02:18:13.425658 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.425539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal" event={"ID":"9df865c4a0e91b8563f787ba1a893c2f","Type":"ContainerStarted","Data":"2a43614555d6640c62b980ab336a687e052a5c589371455c464f6c2f7416f238"}
Jul 07 02:18:13.440893 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.440834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2mlv" event={"ID":"ff16fd4f-d7f3-4591-a4fe-8b8905692255","Type":"ContainerStarted","Data":"4a7fca2adb46f67e76284d280552bb42f52664691088eed4350503a59d171f28"}
Jul 07 02:18:13.465836 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:13.465790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerStarted","Data":"950b3c5940e91314a13182592de79fcdf5c70b8bad44dbab9528270226f885a4"}
Jul 07 02:18:14.278966 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.278933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:14.279440 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.279080    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:14.279523 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.279462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:14.279581 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.279555    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:14.498178 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.498003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal" event={"ID":"9889fb5d6f98348010daa3566e9aa02e","Type":"ContainerStarted","Data":"1ed13e25f56b46b384037de21545fb90f675bba12a02d5e94d385c43a8c31edb"}
Jul 07 02:18:14.514715 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.514656    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-57.ec2.internal" podStartSLOduration=4.514615092 podStartE2EDuration="4.514615092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:13.441011971 +0000 UTC m=+4.748330363" watchObservedRunningTime="2026-07-07 02:18:14.514615092 +0000 UTC m=+5.821933476"
Jul 07 02:18:14.760831 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.760735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:14.760831 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.760813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:14.761083 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.760973    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:14.761083 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.760995    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:14.761083 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.761006    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:14.761083 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.761061    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.761043133 +0000 UTC m=+10.068361522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:14.761285 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.761136    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:14.761285 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.761169    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.761158356 +0000 UTC m=+10.068476742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:14.861805 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:14.861186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:14.861805 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.861364    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:14.861805 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:14.861433    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.861413005 +0000 UTC m=+10.168731375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:15.280606 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:15.280569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:15.281100 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:15.280727    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:16.278487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:16.278454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:16.278704 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:16.278604    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:16.279082 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:16.279064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:16.279196 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:16.279164    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:17.278427 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:17.278383    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:17.278888 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:17.278519    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:18.278329 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:18.278285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:18.278542 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.278445    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:18.279030 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:18.278892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:18.279030 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.278986    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:18.793941 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:18.793893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:18.794124 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:18.793966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:18.794201 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794136    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:18.794201 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794155    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:18.794201 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794168    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:18.794351 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794225    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.79420444 +0000 UTC m=+18.101522828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:18.794670 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794648    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.794823 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.794704    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.794689161 +0000 UTC m=+18.102007533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.894577 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:18.894534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:18.894793 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.894734    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:18.894877 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:18.894797    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.89477226 +0000 UTC m=+18.202090627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:19.280211 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:19.280117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:19.280683 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:19.280236    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:20.278754 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:20.278435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:20.278754 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:20.278591    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:20.278754 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:20.278648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:20.278754 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:20.278721    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:21.280187 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:21.279768    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:21.280187 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:21.279901    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:22.278373 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:22.278333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:22.278568 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:22.278383    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:22.278568 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:22.278468    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:22.278725 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:22.278614    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:23.279176 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:23.279083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:23.279571 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:23.279226    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:24.278949 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:24.278907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:24.279122 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:24.279051    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:24.279423 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:24.278908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:24.279799 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:24.279520    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:25.278162 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:25.278125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:25.278336 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:25.278253    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:26.278290 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:26.278249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:26.278791 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:26.278298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:26.278791 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.278388    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:26.278791 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.278495    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:26.858039 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:26.857999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:26.858039 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:26.858050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:26.858245 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858179    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:26.858245 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858180    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:26.858245 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858195    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:26.858245 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858213    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:26.858398 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858262    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:42.858239487 +0000 UTC m=+34.165557880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:26.858398 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.858282    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:18:42.858271788 +0000 UTC m=+34.165590162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:26.958910 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:26.958865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:26.959073 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.958992    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:26.959073 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:26.959055    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:42.959035833 +0000 UTC m=+34.266354204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:27.278933 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:27.278840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:27.279372 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:27.278990    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:28.278945 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:28.278901    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:28.279342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:28.278918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:28.279342 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:28.279047    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:28.279342 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:28.279082    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:29.281452 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:29.281422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:29.281852 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:29.281520    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:30.279067 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:30.279029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:30.279067 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:30.279048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:30.279381 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:30.279155    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:30.279381 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:30.279264    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:31.278223 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.278190    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:31.278663 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:31.278334    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:31.539930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.539655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerStarted","Data":"94c66894b69a53e8b4c74d56cc1fbbd3c9a04fc84169158b3c7080fe0a3c86f6"}
Jul 07 02:18:31.541198 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.541166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4h5hb" event={"ID":"8f161180-18e0-47b7-af90-2603fa058c4f","Type":"ContainerStarted","Data":"01883e2ce09f31a5e17c5035c7d049ed9066baa196ef37c717fdf6b2f108d4f7"}
Jul 07 02:18:31.543019 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.542993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"1b6404f63c29312d0bde98bc88069c41c9c3331546fa0766a2e5efdc5f32765d"}
Jul 07 02:18:31.543117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.543054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"824a5374dff8f0efda1c018a570ae5819830570a8237bbee917e8c790a0eb8e8"}
Jul 07 02:18:31.544153 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.544131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ngcnf" event={"ID":"d816096e-154e-490a-9c94-815e35733b2f","Type":"ContainerStarted","Data":"4b0c4e4c800cbf53b1d1df5bbe64787833f7dcf7d4c35f091e185d0bd84bb741"}
Jul 07 02:18:31.545398 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.545375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8szf" event={"ID":"0e00c637-d42b-4518-bd76-555484211f93","Type":"ContainerStarted","Data":"48c2d8ccaa38ced6f2ce1ad991f5755b0f6a1c3deec0f207ac6c816895b41256"}
Jul 07 02:18:31.547069 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.547003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q" event={"ID":"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117","Type":"ContainerStarted","Data":"7aa48549722f846fd8c1ec8b762985d2b3f81a02e449494a5e5289de3e4bb51c"}
Jul 07 02:18:31.549072 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.549041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqwgh" event={"ID":"49f46c00-1bba-4491-aa68-aef5b6d3c490","Type":"ContainerStarted","Data":"eb8625f0a9225aedc8106fa53c06338229f1ab4ad0a2d6617f9c82e6d1b8ed04"}
Jul 07 02:18:31.550256 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.550230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2mlv" event={"ID":"ff16fd4f-d7f3-4591-a4fe-8b8905692255","Type":"ContainerStarted","Data":"c492d8047813331e09c76496114fedc96faeab64bddb2b60cb3f760be9b80b2a"}
Jul 07 02:18:31.568936 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.568896    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-57.ec2.internal" podStartSLOduration=21.568881178 podStartE2EDuration="21.568881178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:14.515672386 +0000 UTC m=+5.822990777" watchObservedRunningTime="2026-07-07 02:18:31.568881178 +0000 UTC m=+22.876199567"
Jul 07 02:18:31.588761 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.588699    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gqwgh" podStartSLOduration=3.676656324 podStartE2EDuration="22.588678616s" totalImagesPullingTime="18.912022292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.588038779 +0000 UTC m=+22.895357196" watchObservedRunningTime="2026-07-07 02:18:31.588678616 +0000 UTC m=+22.895997006"
Jul 07 02:18:31.624201 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.624147    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q8szf" podStartSLOduration=3.769460162 podStartE2EDuration="22.624129273s" totalImagesPullingTime="18.854669111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.604714701 +0000 UTC m=+22.912033091" watchObservedRunningTime="2026-07-07 02:18:31.624129273 +0000 UTC m=+22.931447663"
Jul 07 02:18:31.624386 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.624308    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t2mlv" podStartSLOduration=3.750118838 podStartE2EDuration="22.62430308s" totalImagesPullingTime="18.874184242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.623674506 +0000 UTC m=+22.930992897" watchObservedRunningTime="2026-07-07 02:18:31.62430308 +0000 UTC m=+22.931621470"
Jul 07 02:18:31.658095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.658051    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ngcnf" podStartSLOduration=3.78246004 podStartE2EDuration="22.658027482s" totalImagesPullingTime="18.875567442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.63924047 +0000 UTC m=+22.946558860" watchObservedRunningTime="2026-07-07 02:18:31.658027482 +0000 UTC m=+22.965346040"
Jul 07 02:18:31.658247 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:31.658228    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4h5hb" podStartSLOduration=3.803990589 podStartE2EDuration="22.658221873s" totalImagesPullingTime="18.854231284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.6578598 +0000 UTC m=+22.965178191" watchObservedRunningTime="2026-07-07 02:18:31.658221873 +0000 UTC m=+22.965540263"
Jul 07 02:18:32.278265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.278074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:32.279040 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.278120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:32.279040 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:32.278344    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:32.279040 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:32.278456    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:32.554045 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.554011    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="94c66894b69a53e8b4c74d56cc1fbbd3c9a04fc84169158b3c7080fe0a3c86f6" exitCode=0
Jul 07 02:18:32.554225 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.554102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"94c66894b69a53e8b4c74d56cc1fbbd3c9a04fc84169158b3c7080fe0a3c86f6"}
Jul 07 02:18:32.556814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.556793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:18:32.557127 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557101    2561 generic.go:356] "Generic (PLEG): container finished" podID="4756d2f8-25d8-497e-be3e-d934a593ccf4" containerID="1b6404f63c29312d0bde98bc88069c41c9c3331546fa0766a2e5efdc5f32765d" exitCode=1
Jul 07 02:18:32.557247 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerDied","Data":"1b6404f63c29312d0bde98bc88069c41c9c3331546fa0766a2e5efdc5f32765d"}
Jul 07 02:18:32.557304 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"e318b1f3eafb4076aa7cb10333bab932f552d167519fba88831cef1567d141ec"}
Jul 07 02:18:32.557336 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"57ef8393d25c344ee8400c9ccf2411172090d447e28c7231fb4ebcd0e5446ad7"}
Jul 07 02:18:32.557369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"b08f95b5e0ff6ca927c943e71ec587dea37b7196b97078cc388bd03e6f4dd431"}
Jul 07 02:18:32.557404 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.557385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"650b8bf75e72fffe04c05191fc45c3fc2ba9c5ef8c7fa40c599c03bbabf08fdb"}
Jul 07 02:18:32.795076 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:32.795011    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 02:18:33.180908 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.180775    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:32.795031382Z","UUID":"646bc5f5-519a-4714-bf47-ac949dd63f51","Handler":null,"Name":"","Endpoint":""}
Jul 07 02:18:33.182467 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.182435    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 02:18:33.182610 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.182481    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 02:18:33.279132 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.279093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:33.279611 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:33.279251    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:33.560942 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.560910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdnwc" event={"ID":"67ea23be-df66-49b6-8ebc-efb851af5659","Type":"ContainerStarted","Data":"b9f1fef15f6451b68454db1c373683e205dc0e51cbdfdb455f89da55b055a64c"}
Jul 07 02:18:33.562880 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.562852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q" event={"ID":"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117","Type":"ContainerStarted","Data":"c272acc4612db53c87c2947000749aa5224d9375bdb4de456b14ce5ebd3a0228"}
Jul 07 02:18:33.577468 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.577413    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wdnwc" podStartSLOduration=5.725597447 podStartE2EDuration="24.577395274s" totalImagesPullingTime="18.851797827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:33.576737037 +0000 UTC m=+24.884055431" watchObservedRunningTime="2026-07-07 02:18:33.577395274 +0000 UTC m=+24.884713663"
Jul 07 02:18:33.645469 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:33.645438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t2mlv_ff16fd4f-d7f3-4591-a4fe-8b8905692255/dns-node-resolver/0.log"
Jul 07 02:18:34.278830 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.278734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:34.279008 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.278740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:34.279008 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:34.278869    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:34.279008 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:34.278999    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:34.568939 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.568909    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:18:34.569381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.569332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"3ef9364bb1448d66df4550791cae3b31f14270122654d93c6efc829e4fee5e81"}
Jul 07 02:18:34.571759 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.571729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q" event={"ID":"b1c05da0-30a5-4cd3-a6f4-1e4c5201d117","Type":"ContainerStarted","Data":"b6b41319e1ec53a87b57a4da9d983217265c7c599272e47814f5c419213fae3e"}
Jul 07 02:18:34.821657 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:34.821564    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ngcnf_d816096e-154e-490a-9c94-815e35733b2f/node-ca/0.log"
Jul 07 02:18:35.278567 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:35.278484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:35.278732 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:35.278638    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:36.278734 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.278513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:36.279194 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.278567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:36.279194 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:36.278854    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:36.279194 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:36.278981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:36.519466 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.519375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:36.520176 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.520150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:36.535550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.535496    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vj9q" podStartSLOduration=6.369941966 podStartE2EDuration="27.535473379s" totalImagesPullingTime="21.165531413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:34.598309345 +0000 UTC m=+25.905627734" watchObservedRunningTime="2026-07-07 02:18:36.535473379 +0000 UTC m=+27.842791776"
Jul 07 02:18:36.576932 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.576899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:36.577432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:36.577411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8szf"
Jul 07 02:18:37.278375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.278120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:37.278552 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:37.278418    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:37.581440 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.581412    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:18:37.582372 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.581767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"19835f3c3d0576690f81d9a989a9150c90a20c2720205c77fe3cc42ae6fcefe2"}
Jul 07 02:18:37.582372 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.582008    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:37.582372 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.582322    2561 scope.go:117] "RemoveContainer" containerID="1b6404f63c29312d0bde98bc88069c41c9c3331546fa0766a2e5efdc5f32765d"
Jul 07 02:18:37.584026 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.583951    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="6a120214f8360b096544ba0f8fb9e185683beb420997dd0048bf616114449c94" exitCode=0
Jul 07 02:18:37.584026 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.583993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"6a120214f8360b096544ba0f8fb9e185683beb420997dd0048bf616114449c94"}
Jul 07 02:18:37.599238 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:37.599214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:38.279184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.279150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:38.279442 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:38.279279    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:38.279442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.279327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:38.279442 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:38.279433    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:38.591171 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.591146    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:18:38.592079 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.592060    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 02:18:38.592894 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.592871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" event={"ID":"4756d2f8-25d8-497e-be3e-d934a593ccf4","Type":"ContainerStarted","Data":"fb466c4b0bc6f58b72f04cb357fe2c6bad6f081bbe17049e238f43c4dabdbf74"}
Jul 07 02:18:38.593071 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.593055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:38.611595 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.611566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:38.624862 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.624806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km" podStartSLOduration=10.693319329 podStartE2EDuration="29.624788991s" totalImagesPullingTime="18.931469662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:38.624518949 +0000 UTC m=+29.931837366" watchObservedRunningTime="2026-07-07 02:18:38.624788991 +0000 UTC m=+29.932107380"
Jul 07 02:18:38.744932 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:38.744896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:18:39.280193 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:39.280156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:39.280371 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:39.280276    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:39.596081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:39.596045    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="ae05fa9b3cd0be911ba09338246f8a67b848932edad61a14fa46298f29b71a1f" exitCode=0
Jul 07 02:18:39.596553 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:39.596130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"ae05fa9b3cd0be911ba09338246f8a67b848932edad61a14fa46298f29b71a1f"}
Jul 07 02:18:40.279032 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:40.279000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:40.279206 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:40.278998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:40.279206 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:40.279125    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:40.279206 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:40.279179    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:40.600941 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:40.600906    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="1e77e7ee7ab59d8b069dbd3acac74977326136f16e87d4af1e3ba6df1ef4c840" exitCode=0
Jul 07 02:18:40.601388 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:40.600993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"1e77e7ee7ab59d8b069dbd3acac74977326136f16e87d4af1e3ba6df1ef4c840"}
Jul 07 02:18:41.278772 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:41.278729    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:41.279005 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:41.278861    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:42.278660 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:42.278603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:42.278660 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:42.278658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:42.279228 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.278793    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:42.279228 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.278925    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:42.873317 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:42.873276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:42.873341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873445    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873469    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873486    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873498    2561 projected.go:196] Error preparing data for projected volume kube-api-access-t92t9 for pod openshift-network-diagnostics/network-check-target-r2x82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:42.873523 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873512    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret podName:fb558531-20cf-4b45-a955-5f1ca96e1516 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:14.873496304 +0000 UTC m=+66.180814685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret") pod "global-pull-secret-syncer-dqm7c" (UID: "fb558531-20cf-4b45-a955-5f1ca96e1516") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:42.873813 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.873542    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9 podName:d1ddd8ce-320b-4115-a3b5-d7082fe3da1a nodeName:}" failed. No retries permitted until 2026-07-07 02:19:14.873527908 +0000 UTC m=+66.180846277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t92t9" (UniqueName: "kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9") pod "network-check-target-r2x82" (UID: "d1ddd8ce-320b-4115-a3b5-d7082fe3da1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:42.974520 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:42.974479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:42.974737 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.974664    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:42.974816 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:42.974744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs podName:ed4f47c0-1a2e-4045-826a-65ba949859d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:14.974722113 +0000 UTC m=+66.282040485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs") pod "network-metrics-daemon-t5bd7" (UID: "ed4f47c0-1a2e-4045-826a-65ba949859d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:43.278859 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:43.278772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:43.279299 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:43.278886    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:44.278127 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:44.278089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:44.278344 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:44.278221    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:44.278344 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:44.278274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:44.278459 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:44.278369    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:45.279258 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:45.279180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:45.279662 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:45.279298    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:45.616243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:45.616206    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="10be772cd6fc23f2d355d7010bc1407ad52e0dea977800b8abf73c4bac4e26e9" exitCode=0
Jul 07 02:18:45.616243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:45.616248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"10be772cd6fc23f2d355d7010bc1407ad52e0dea977800b8abf73c4bac4e26e9"}
Jul 07 02:18:46.279065 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:46.279024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:46.279065 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:46.279058    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:46.279315 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:46.279137    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:46.279315 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:46.279279    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:46.621448 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:46.621412    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3172c72-6087-4af8-bf4f-24475348408a" containerID="ae6bdd45cb57224c3f749bb77357c11b264060493b68be0f332b830d7ef16d3d" exitCode=0
Jul 07 02:18:46.621448 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:46.621451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerDied","Data":"ae6bdd45cb57224c3f749bb77357c11b264060493b68be0f332b830d7ef16d3d"}
Jul 07 02:18:47.278479 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:47.278442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:47.278679 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:47.278558    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:47.626847 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:47.626812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jngg" event={"ID":"d3172c72-6087-4af8-bf4f-24475348408a","Type":"ContainerStarted","Data":"f91a507f952db885e5726439e0c2c339810afc29a36a7a7fb45d4608b5d3f01f"}
Jul 07 02:18:47.652175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:47.652118    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6jngg" podStartSLOduration=9.59591388 podStartE2EDuration="38.652101569s" totalImagesPullingTime="29.056187689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:47.650492292 +0000 UTC m=+38.957810682" watchObservedRunningTime="2026-07-07 02:18:47.652101569 +0000 UTC m=+38.959419958"
Jul 07 02:18:48.278765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:48.278724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:48.278955 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:48.278740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:48.278955 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:48.278840    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:48.278955 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:48.278910    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:49.280064 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:49.279855    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:49.280558 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:49.280121    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:50.278589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:50.278548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:50.278589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:50.278588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:50.278897 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:50.278709    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:50.278897 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:50.278800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:51.279015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:51.278980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:51.279405 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:51.279091    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:52.278080 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:52.278038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:52.278080 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:52.278077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:52.278374 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:52.278178    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:52.278374 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:52.278287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:53.278823 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:53.278789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:53.279246 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:53.278900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:54.278284 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:54.278246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:54.278496 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:54.278247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:54.278496 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:54.278386    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:54.278496 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:54.278432    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:55.278394 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:55.278356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:55.278855 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:55.278514    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:56.278942 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:56.278902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:56.279484 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:56.279040    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:56.279484 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:56.279090    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:56.279484 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:56.279187    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:57.278875 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:57.278839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:57.279073 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:57.278977    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:58.278160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.278123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:58.278355 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.278169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:58.278355 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:58.278241    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:18:58.278491 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:58.278370    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:58.857319 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.857192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r2x82"]
Jul 07 02:18:58.858163 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.857543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:18:58.858163 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:58.857695    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:18:58.858163 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.858102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqm7c"]
Jul 07 02:18:58.858318 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.858217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:18:58.858368 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:58.858327    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:18:58.884960 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.884929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5bd7"]
Jul 07 02:18:58.885129 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:18:58.885070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:18:58.885321 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:18:58.885289    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:19:00.278679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:00.278642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:00.279171 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:00.278759    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:19:01.278903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:01.278865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:19:01.279300 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:01.278986    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5bd7" podUID="ed4f47c0-1a2e-4045-826a-65ba949859d3"
Jul 07 02:19:01.279300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:01.279038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:19:01.279300 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:01.279128    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqm7c" podUID="fb558531-20cf-4b45-a955-5f1ca96e1516"
Jul 07 02:19:02.278555 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:02.278517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:02.278775 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:02.278672    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2x82" podUID="d1ddd8ce-320b-4115-a3b5-d7082fe3da1a"
Jul 07 02:19:02.965301 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:02.965115    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-57.ec2.internal" event="NodeReady"
Jul 07 02:19:02.965679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:02.965408    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 02:19:03.011161 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.011123    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jtz8x"]
Jul 07 02:19:03.013412 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.013378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.014792 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.014765    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dlvjt"]
Jul 07 02:19:03.016766 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.016744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.017289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.017264    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4v9ss\""
Jul 07 02:19:03.017519 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.017498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 02:19:03.017610 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.017548    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 02:19:03.019133 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.019107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 02:19:03.019283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.019108    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 02:19:03.019283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.019231    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 02:19:03.019442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.019387    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 02:19:03.019545 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.019525    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vjbfk\""
Jul 07 02:19:03.026010 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.025986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtz8x"]
Jul 07 02:19:03.032090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.032059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlvjt"]
Jul 07 02:19:03.118338 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.118299    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rlhqp"]
Jul 07 02:19:03.119733 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c85221-43cb-4ca4-94b1-86a895e8c648-config-volume\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.119877 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0823e9dd-da0a-4311-b643-5519fcf57046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.119877 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c85221-43cb-4ca4-94b1-86a895e8c648-tmp-dir\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.119877 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85221-43cb-4ca4-94b1-86a895e8c648-metrics-tls\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.119989 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0823e9dd-da0a-4311-b643-5519fcf57046-crio-socket\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.119989 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.119912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nnq\" (UniqueName: \"kubernetes.io/projected/0823e9dd-da0a-4311-b643-5519fcf57046-kube-api-access-d5nnq\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.120105 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.120007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0823e9dd-da0a-4311-b643-5519fcf57046-data-volume\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.120105 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.120029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzp8p\" (UniqueName: \"kubernetes.io/projected/f2c85221-43cb-4ca4-94b1-86a895e8c648-kube-api-access-nzp8p\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.120105 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.120046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0823e9dd-da0a-4311-b643-5519fcf57046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.120300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.120287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.123026 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.123010    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 02:19:03.123132 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.123115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 02:19:03.123175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.123148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tp7rb\""
Jul 07 02:19:03.123453 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.123441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 02:19:03.129903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.129880    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rlhqp"]
Jul 07 02:19:03.220979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.220882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85221-43cb-4ca4-94b1-86a895e8c648-metrics-tls\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.220979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.220927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0823e9dd-da0a-4311-b643-5519fcf57046-crio-socket\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.220979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.220947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nnq\" (UniqueName: \"kubernetes.io/projected/0823e9dd-da0a-4311-b643-5519fcf57046-kube-api-access-d5nnq\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.220979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.220971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0823e9dd-da0a-4311-b643-5519fcf57046-data-volume\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec88dc48-87af-434b-b851-19eb2e8d449f-cert\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp8p\" (UniqueName: \"kubernetes.io/projected/f2c85221-43cb-4ca4-94b1-86a895e8c648-kube-api-access-nzp8p\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0823e9dd-da0a-4311-b643-5519fcf57046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c85221-43cb-4ca4-94b1-86a895e8c648-config-volume\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0823e9dd-da0a-4311-b643-5519fcf57046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221153    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0823e9dd-da0a-4311-b643-5519fcf57046-crio-socket\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.221303 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlgh\" (UniqueName: \"kubernetes.io/projected/ec88dc48-87af-434b-b851-19eb2e8d449f-kube-api-access-pwlgh\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.221655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c85221-43cb-4ca4-94b1-86a895e8c648-tmp-dir\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.221655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221349    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0823e9dd-da0a-4311-b643-5519fcf57046-data-volume\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.221655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221578    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c85221-43cb-4ca4-94b1-86a895e8c648-tmp-dir\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.221845 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221823    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c85221-43cb-4ca4-94b1-86a895e8c648-config-volume\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.221907 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.221848    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0823e9dd-da0a-4311-b643-5519fcf57046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.225246 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.225223    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85221-43cb-4ca4-94b1-86a895e8c648-metrics-tls\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.225365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.225281    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0823e9dd-da0a-4311-b643-5519fcf57046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.229143 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.229117    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nnq\" (UniqueName: \"kubernetes.io/projected/0823e9dd-da0a-4311-b643-5519fcf57046-kube-api-access-d5nnq\") pod \"insights-runtime-extractor-dlvjt\" (UID: \"0823e9dd-da0a-4311-b643-5519fcf57046\") " pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.229273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.229126    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp8p\" (UniqueName: \"kubernetes.io/projected/f2c85221-43cb-4ca4-94b1-86a895e8c648-kube-api-access-nzp8p\") pod \"dns-default-jtz8x\" (UID: \"f2c85221-43cb-4ca4-94b1-86a895e8c648\") " pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.278169 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.278139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:19:03.278328 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.278180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:19:03.281238 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.281217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:19:03.281238 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.281235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kthkt\""
Jul 07 02:19:03.281415 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.281219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:19:03.321662 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.321606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec88dc48-87af-434b-b851-19eb2e8d449f-cert\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.321847 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.321705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlgh\" (UniqueName: \"kubernetes.io/projected/ec88dc48-87af-434b-b851-19eb2e8d449f-kube-api-access-pwlgh\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.324327 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.324301    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec88dc48-87af-434b-b851-19eb2e8d449f-cert\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.326036 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.326014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:03.330399 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.330375    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlgh\" (UniqueName: \"kubernetes.io/projected/ec88dc48-87af-434b-b851-19eb2e8d449f-kube-api-access-pwlgh\") pod \"ingress-canary-rlhqp\" (UID: \"ec88dc48-87af-434b-b851-19eb2e8d449f\") " pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.335167 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.335142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlvjt"
Jul 07 02:19:03.429784 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.429755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rlhqp"
Jul 07 02:19:03.480495 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.480351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlvjt"]
Jul 07 02:19:03.483230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.483199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtz8x"]
Jul 07 02:19:03.488922 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:03.488886    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0823e9dd_da0a_4311_b643_5519fcf57046.slice/crio-67a9fa0681d5c2a7298494867f1f2cdedd4d0eaa386b8b9e5a040d0c2c4d1d4f WatchSource:0}: Error finding container 67a9fa0681d5c2a7298494867f1f2cdedd4d0eaa386b8b9e5a040d0c2c4d1d4f: Status 404 returned error can't find the container with id 67a9fa0681d5c2a7298494867f1f2cdedd4d0eaa386b8b9e5a040d0c2c4d1d4f
Jul 07 02:19:03.490120 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:03.490071    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c85221_43cb_4ca4_94b1_86a895e8c648.slice/crio-0af5b0770051ccc9bdbcb3540f41cef244c41e86641f766f4462fb1d1bda4881 WatchSource:0}: Error finding container 0af5b0770051ccc9bdbcb3540f41cef244c41e86641f766f4462fb1d1bda4881: Status 404 returned error can't find the container with id 0af5b0770051ccc9bdbcb3540f41cef244c41e86641f766f4462fb1d1bda4881
Jul 07 02:19:03.568085 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.568039    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rlhqp"]
Jul 07 02:19:03.572174 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:03.572134    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec88dc48_87af_434b_b851_19eb2e8d449f.slice/crio-5308c9e20ed75df3e78699df34548ae925e5f534dafdfd67e016198c4255efcd WatchSource:0}: Error finding container 5308c9e20ed75df3e78699df34548ae925e5f534dafdfd67e016198c4255efcd: Status 404 returned error can't find the container with id 5308c9e20ed75df3e78699df34548ae925e5f534dafdfd67e016198c4255efcd
Jul 07 02:19:03.664263 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.664229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtz8x" event={"ID":"f2c85221-43cb-4ca4-94b1-86a895e8c648","Type":"ContainerStarted","Data":"0af5b0770051ccc9bdbcb3540f41cef244c41e86641f766f4462fb1d1bda4881"}
Jul 07 02:19:03.665204 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.665170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rlhqp" event={"ID":"ec88dc48-87af-434b-b851-19eb2e8d449f","Type":"ContainerStarted","Data":"5308c9e20ed75df3e78699df34548ae925e5f534dafdfd67e016198c4255efcd"}
Jul 07 02:19:03.666243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:03.666226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlvjt" event={"ID":"0823e9dd-da0a-4311-b643-5519fcf57046","Type":"ContainerStarted","Data":"67a9fa0681d5c2a7298494867f1f2cdedd4d0eaa386b8b9e5a040d0c2c4d1d4f"}
Jul 07 02:19:04.279695 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:04.278672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:04.282074 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:04.281778    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:19:04.283400 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:04.282685    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv29t\""
Jul 07 02:19:04.283400 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:04.282936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:19:08.206557 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.206476    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"]
Jul 07 02:19:08.214549 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.214522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.218889 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.218853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 02:19:08.219080 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.218861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 02:19:08.220078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.219829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 02:19:08.220078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.219848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 02:19:08.220078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.219860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 02:19:08.220078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.219935    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hx2xt\""
Jul 07 02:19:08.221741 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.221718    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"]
Jul 07 02:19:08.223966 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.223946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.227915 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.227892    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 02:19:08.228201 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.227937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 02:19:08.228283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.227973    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7rhjp\""
Jul 07 02:19:08.228283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.228050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 02:19:08.233549 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.232908    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"]
Jul 07 02:19:08.243542 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.243461    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"]
Jul 07 02:19:08.247345 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.247317    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dqpwp"]
Jul 07 02:19:08.251016 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.250994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.254572 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.254545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 02:19:08.254941 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.254919    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 02:19:08.255395 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.255288    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 02:19:08.256798 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.256774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8twrq\""
Jul 07 02:19:08.361616 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.361616 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-wtmp\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-root\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-textfile\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.361866 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8312a6ec-7348-4770-baec-9bab8b97eb2f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8789e80-7a85-4faf-be6a-12a4bfeb2491-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.361987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqgz\" (UniqueName: \"kubernetes.io/projected/8312a6ec-7348-4770-baec-9bab8b97eb2f-kube-api-access-mgqgz\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-sys\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72mf\" (UniqueName: \"kubernetes.io/projected/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-api-access-f72mf\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.362180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.362519 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2tk\" (UniqueName: \"kubernetes.io/projected/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-kube-api-access-4c2tk\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.362519 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.362207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-metrics-client-ca\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.463264 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8312a6ec-7348-4770-baec-9bab8b97eb2f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.463264 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8789e80-7a85-4faf-be6a-12a4bfeb2491-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.463496 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.463496 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqgz\" (UniqueName: \"kubernetes.io/projected/8312a6ec-7348-4770-baec-9bab8b97eb2f-kube-api-access-mgqgz\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.463722 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463700    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8789e80-7a85-4faf-be6a-12a4bfeb2491-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.463809 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.463809 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-sys\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.463917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72mf\" (UniqueName: \"kubernetes.io/projected/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-api-access-f72mf\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.463917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.463917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.463917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463886    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-sys\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.463917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2tk\" (UniqueName: \"kubernetes.io/projected/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-kube-api-access-4c2tk\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-metrics-client-ca\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463978    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8312a6ec-7348-4770-baec-9bab8b97eb2f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.463991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-wtmp\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464075    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-root\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464133    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls podName:8312a6ec-7348-4770-baec-9bab8b97eb2f nodeName:}" failed. No retries permitted until 2026-07-07 02:19:08.964116053 +0000 UTC m=+60.271434438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-z8nbk" (UID: "8312a6ec-7348-4770-baec-9bab8b97eb2f") : secret "openshift-state-metrics-tls" not found
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-textfile\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464187    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464235    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls podName:b8789e80-7a85-4faf-be6a-12a4bfeb2491 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:08.964220232 +0000 UTC m=+60.271538601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-dmsjx" (UID: "b8789e80-7a85-4faf-be6a-12a4bfeb2491") : secret "kube-state-metrics-tls" not found
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464238    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-wtmp\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.464289 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464292    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-root\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464232    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464450    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464575    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-metrics-client-ca\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.464595    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-textfile\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464760    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.464852    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls podName:2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a nodeName:}" failed. No retries permitted until 2026-07-07 02:19:08.964840816 +0000 UTC m=+60.272159183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls") pod "node-exporter-dqpwp" (UID: "2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a") : secret "node-exporter-tls" not found
Jul 07 02:19:08.465377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.465249    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.467163 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.467137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.467305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.467284    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.467362 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.467343    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.472133 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.472075    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqgz\" (UniqueName: \"kubernetes.io/projected/8312a6ec-7348-4770-baec-9bab8b97eb2f-kube-api-access-mgqgz\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.473403 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.473379    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2tk\" (UniqueName: \"kubernetes.io/projected/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-kube-api-access-4c2tk\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.473608 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.473586    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72mf\" (UniqueName: \"kubernetes.io/projected/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-api-access-f72mf\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.884709 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.884677    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:08.889605 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.887493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.890188 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.890166    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mmk7l\""
Jul 07 02:19:08.890278 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.890166    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 02:19:08.891507 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891311    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 02:19:08.891507 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891346    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 02:19:08.891507 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 02:19:08.891743 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891512    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 02:19:08.891743 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891648    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 02:19:08.891743 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.891660    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 02:19:08.897373 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.897180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 02:19:08.899294 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.899270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:08.970889 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.970824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.970889 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.970890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjpb\" (UniqueName: \"kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.970969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.970998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971145    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 02:19:08.971175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971208    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls podName:8312a6ec-7348-4770-baec-9bab8b97eb2f nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.971187421 +0000 UTC m=+61.278505789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-z8nbk" (UID: "8312a6ec-7348-4770-baec-9bab8b97eb2f") : secret "openshift-state-metrics-tls" not found
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971264    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971311    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls podName:b8789e80-7a85-4faf-be6a-12a4bfeb2491 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.971296627 +0000 UTC m=+61.278615012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-dmsjx" (UID: "b8789e80-7a85-4faf-be6a-12a4bfeb2491") : secret "kube-state-metrics-tls" not found
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971337    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:08.971349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:08.971477 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:08.971372    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls podName:2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.971359996 +0000 UTC m=+61.278678368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls") pod "node-exporter-dqpwp" (UID: "2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a") : secret "node-exporter-tls" not found
Jul 07 02:19:09.072577 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjpb\" (UniqueName: \"kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072815 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072815 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072815 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.072992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.072912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.073531 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.073508    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.073660 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.073505    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.073660 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.073583    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.074149 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.074128    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.075662 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.075641    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.076142 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.076115    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.083149 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.083121    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjpb\" (UniqueName: \"kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb\") pod \"console-8d9c69899-dpl7z\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") " pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.201783 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.201757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mmk7l\""
Jul 07 02:19:09.209830 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.209809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:09.289435 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.289408    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:19:09.292731 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.292707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.295790 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.295533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 02:19:09.295901 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.295884    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 02:19:09.297329 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.296753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 02:19:09.297329 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.297013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 02:19:09.297329 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.297230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 02:19:09.297472 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.297456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 02:19:09.297723 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.297709    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 02:19:09.297949 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.297934    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 02:19:09.298153 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.298139    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 02:19:09.299671 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.299653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t5lx7\""
Jul 07 02:19:09.309188 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.309166    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:19:09.357531 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.357444    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:09.361702 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:09.361676    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1178fbe8_d6ea_42c6_a885_3d282a608cde.slice/crio-c6a6a5fddf5034717289d55071f5fa44a1a50861f5f068d28c852601ab55c95b WatchSource:0}: Error finding container c6a6a5fddf5034717289d55071f5fa44a1a50861f5f068d28c852601ab55c95b: Status 404 returned error can't find the container with id c6a6a5fddf5034717289d55071f5fa44a1a50861f5f068d28c852601ab55c95b
Jul 07 02:19:09.377650 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377771 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377771 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377771 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377771 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.377943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.378122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.378122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.377993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.378122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.378021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.378122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.378061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxxn\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479189 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxxn\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479381 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:09.479315    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:09.479396    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls podName:1160d5d2-4d22-496a-bcb3-3b23bff0f402 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.979373858 +0000 UTC m=+61.286692243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402") : secret "alertmanager-main-tls" not found
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:19:09.479478    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle podName:1160d5d2-4d22-496a-bcb3-3b23bff0f402 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.979458651 +0000 UTC m=+61.286777034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402") : configmap references non-existent config key: ca-bundle.crt
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.479778 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.479759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.480179 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.480148    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.481346 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.481010    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482576 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482523    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482576 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482529    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482742 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482691    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482816 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482784    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482862 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482802    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.482898 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.482864    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.483133 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.483118    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.483590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.483569    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.495150 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.495123    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxxn\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.686845 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.686738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rlhqp" event={"ID":"ec88dc48-87af-434b-b851-19eb2e8d449f","Type":"ContainerStarted","Data":"590a6f5139ca28871fd52add8d56d5594cf87ea00589a69b5d005d413d2b054d"}
Jul 07 02:19:09.688284 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.688248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlvjt" event={"ID":"0823e9dd-da0a-4311-b643-5519fcf57046","Type":"ContainerStarted","Data":"b04fcd8dc8d85cdbff1600da06ac9346299a65c052323f8518642a805756c075"}
Jul 07 02:19:09.690452 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.690427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtz8x" event={"ID":"f2c85221-43cb-4ca4-94b1-86a895e8c648","Type":"ContainerStarted","Data":"392080660eb48c8cff92308169ed8fd93f7774955fe6b8d8a17444fe0e84d16e"}
Jul 07 02:19:09.690565 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.690477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtz8x" event={"ID":"f2c85221-43cb-4ca4-94b1-86a895e8c648","Type":"ContainerStarted","Data":"7769befd7ecc0371b7cd96fe1dd96c8bf3f7ef4ce4ee80bb74721cf884857f92"}
Jul 07 02:19:09.690565 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.690562    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:09.691767 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.691743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9c69899-dpl7z" event={"ID":"1178fbe8-d6ea-42c6-a885-3d282a608cde","Type":"ContainerStarted","Data":"c6a6a5fddf5034717289d55071f5fa44a1a50861f5f068d28c852601ab55c95b"}
Jul 07 02:19:09.704927 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.704870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rlhqp" podStartSLOduration=1.465868312 podStartE2EDuration="6.704851617s" totalImagesPullingTime="5.238983305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:09.704315719 +0000 UTC m=+61.011634133" watchObservedRunningTime="2026-07-07 02:19:09.704851617 +0000 UTC m=+61.012170009"
Jul 07 02:19:09.722541 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.722481    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jtz8x" podStartSLOduration=2.423697567 podStartE2EDuration="7.722456901s" totalImagesPullingTime="5.298759334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:09.720988632 +0000 UTC m=+61.028307023" watchObservedRunningTime="2026-07-07 02:19:09.722456901 +0000 UTC m=+61.029775292"
Jul 07 02:19:09.985378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.985273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:09.985378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.985324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:09.985613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.985397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.985613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.985454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.985613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.985510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:09.987114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.987064    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.988733 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.988662    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8312a6ec-7348-4770-baec-9bab8b97eb2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z8nbk\" (UID: \"8312a6ec-7348-4770-baec-9bab8b97eb2f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:09.988733 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.988711    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8789e80-7a85-4faf-be6a-12a4bfeb2491-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dmsjx\" (UID: \"b8789e80-7a85-4faf-be6a-12a4bfeb2491\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:09.988869 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.988711    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:09.988869 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:09.988791    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a-node-exporter-tls\") pod \"node-exporter-dqpwp\" (UID: \"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a\") " pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:10.029117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.029081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hx2xt\""
Jul 07 02:19:10.037562 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.037535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"
Jul 07 02:19:10.038928 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.038906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7rhjp\""
Jul 07 02:19:10.047362 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.047331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"
Jul 07 02:19:10.066651 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.066600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8twrq\""
Jul 07 02:19:10.074927 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.074810    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqpwp"
Jul 07 02:19:10.087381 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:10.087347    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7b5a23_4df7_4b1c_ae5c_3ac32c3ad22a.slice/crio-fabbe1a4329cb5a68d2cdea19fc28bcdf45aae5ba3d7891f5c9a96b04eda86b0 WatchSource:0}: Error finding container fabbe1a4329cb5a68d2cdea19fc28bcdf45aae5ba3d7891f5c9a96b04eda86b0: Status 404 returned error can't find the container with id fabbe1a4329cb5a68d2cdea19fc28bcdf45aae5ba3d7891f5c9a96b04eda86b0
Jul 07 02:19:10.198306 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.198272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk"]
Jul 07 02:19:10.202763 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:10.202719    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8312a6ec_7348_4770_baec_9bab8b97eb2f.slice/crio-0756b61b6eae3776d7c865d7254f0903970c864d4f496282f7fa70b2b890b7a5 WatchSource:0}: Error finding container 0756b61b6eae3776d7c865d7254f0903970c864d4f496282f7fa70b2b890b7a5: Status 404 returned error can't find the container with id 0756b61b6eae3776d7c865d7254f0903970c864d4f496282f7fa70b2b890b7a5
Jul 07 02:19:10.205430 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.205409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:19:10.219724 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.219645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx"]
Jul 07 02:19:10.223242 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:10.223093    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8789e80_7a85_4faf_be6a_12a4bfeb2491.slice/crio-ce52c879efb1f0c6de2ff792817dafe289ee6ec2ad277f9cb54e27469dfc8431 WatchSource:0}: Error finding container ce52c879efb1f0c6de2ff792817dafe289ee6ec2ad277f9cb54e27469dfc8431: Status 404 returned error can't find the container with id ce52c879efb1f0c6de2ff792817dafe289ee6ec2ad277f9cb54e27469dfc8431
Jul 07 02:19:10.392246 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.392210    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:19:10.398947 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:10.398896    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1160d5d2_4d22_496a_bcb3_3b23bff0f402.slice/crio-3b964eb8192cec791932f4bd69fc064510ac3532f7251ea9e645d65d74732294 WatchSource:0}: Error finding container 3b964eb8192cec791932f4bd69fc064510ac3532f7251ea9e645d65d74732294: Status 404 returned error can't find the container with id 3b964eb8192cec791932f4bd69fc064510ac3532f7251ea9e645d65d74732294
Jul 07 02:19:10.613873 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.613837    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9q4km"
Jul 07 02:19:10.702789 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.702723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"3b964eb8192cec791932f4bd69fc064510ac3532f7251ea9e645d65d74732294"}
Jul 07 02:19:10.705363 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.705296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqpwp" event={"ID":"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a","Type":"ContainerStarted","Data":"fabbe1a4329cb5a68d2cdea19fc28bcdf45aae5ba3d7891f5c9a96b04eda86b0"}
Jul 07 02:19:10.706887 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.706823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx" event={"ID":"b8789e80-7a85-4faf-be6a-12a4bfeb2491","Type":"ContainerStarted","Data":"ce52c879efb1f0c6de2ff792817dafe289ee6ec2ad277f9cb54e27469dfc8431"}
Jul 07 02:19:10.711572 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.711524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlvjt" event={"ID":"0823e9dd-da0a-4311-b643-5519fcf57046","Type":"ContainerStarted","Data":"db804045110076b5746de71d53e2dbd7256dc74b22779e119914c6a99990b3cb"}
Jul 07 02:19:10.714681 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.714649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk" event={"ID":"8312a6ec-7348-4770-baec-9bab8b97eb2f","Type":"ContainerStarted","Data":"385d318a88e8fb0c6abc03b7d01c2bc9e18c032ce56375b86d9dbbae1d79b1ed"}
Jul 07 02:19:10.714825 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.714702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk" event={"ID":"8312a6ec-7348-4770-baec-9bab8b97eb2f","Type":"ContainerStarted","Data":"7ac7c1e53a51a57535ddc73dc3589833c53600039ebb69073cc8544a3d46f3ff"}
Jul 07 02:19:10.714825 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:10.714731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk" event={"ID":"8312a6ec-7348-4770-baec-9bab8b97eb2f","Type":"ContainerStarted","Data":"0756b61b6eae3776d7c865d7254f0903970c864d4f496282f7fa70b2b890b7a5"}
Jul 07 02:19:11.216195 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.216156    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"]
Jul 07 02:19:11.219287 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.219255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.225396 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.225368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 02:19:11.225983 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.225669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rqkjx\""
Jul 07 02:19:11.225983 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.225875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 02:19:11.226802 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.226781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3ntnm7nrjrj5a\""
Jul 07 02:19:11.227773 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.227752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 02:19:11.227997 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.227979    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 02:19:11.228212 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.228194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 02:19:11.235668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.235648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"]
Jul 07 02:19:11.299192 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbf87aa6-80d0-4e04-b674-3345f5a61a64-metrics-client-ca\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299192 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rptq\" (UniqueName: \"kubernetes.io/projected/dbf87aa6-80d0-4e04-b674-3345f5a61a64-kube-api-access-5rptq\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.299462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.299424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-grpc-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.400710 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rptq\" (UniqueName: \"kubernetes.io/projected/dbf87aa6-80d0-4e04-b674-3345f5a61a64-kube-api-access-5rptq\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.400884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.400884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.400884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.400884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.401101 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-grpc-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.401101 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.400976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbf87aa6-80d0-4e04-b674-3345f5a61a64-metrics-client-ca\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.401101 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.401005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.403124 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.403092    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbf87aa6-80d0-4e04-b674-3345f5a61a64-metrics-client-ca\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.409373 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.409336    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.413432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.413396    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.413812 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.413616    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.415184 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.415141    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.419550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.419494    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rptq\" (UniqueName: \"kubernetes.io/projected/dbf87aa6-80d0-4e04-b674-3345f5a61a64-kube-api-access-5rptq\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.421714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.421687    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-thanos-querier-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.424467 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.422803    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbf87aa6-80d0-4e04-b674-3345f5a61a64-secret-grpc-tls\") pod \"thanos-querier-66d5b8bbf6-dfk87\" (UID: \"dbf87aa6-80d0-4e04-b674-3345f5a61a64\") " pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.534408 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.534368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:11.723102 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.723065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqpwp" event={"ID":"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a","Type":"ContainerStarted","Data":"4a9b80a4f4549cd6ec45e5682aa16b86f73eda7063cb7743661e65fb6f2d8cdc"}
Jul 07 02:19:11.738175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:11.738145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"]
Jul 07 02:19:12.631378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.631347    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f7d695f6f-5txkx"]
Jul 07 02:19:12.633882 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.633858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.636574 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.636538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 02:19:12.636705 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.636572    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 02:19:12.636705 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.636538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 02:19:12.637604 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.637587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 02:19:12.637734 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.637638    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cqd7j\""
Jul 07 02:19:12.638442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.638423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-71vm45rn9jksp\""
Jul 07 02:19:12.644554 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.644532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f7d695f6f-5txkx"]
Jul 07 02:19:12.715287 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-metrics-server-audit-profiles\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-tls\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-client-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f1c00182-20c8-4aec-ab43-749702f26fed-audit-log\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715700 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-client-certs\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.715700 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.715572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj75p\" (UniqueName: \"kubernetes.io/projected/f1c00182-20c8-4aec-ab43-749702f26fed-kube-api-access-qj75p\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.816612 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.816566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-metrics-server-audit-profiles\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.816805 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.816725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-tls\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.816805 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.816762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-client-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.816904 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.816832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.817683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f1c00182-20c8-4aec-ab43-749702f26fed-audit-log\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.817683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-client-certs\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.817683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj75p\" (UniqueName: \"kubernetes.io/projected/f1c00182-20c8-4aec-ab43-749702f26fed-kube-api-access-qj75p\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.817683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817453    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.817683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817567    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f1c00182-20c8-4aec-ab43-749702f26fed-audit-log\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.818121 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.817747    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f1c00182-20c8-4aec-ab43-749702f26fed-metrics-server-audit-profiles\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.820212 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.820184    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-tls\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.820313 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.820222    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-secret-metrics-server-client-certs\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.820718 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.820698    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f1c00182-20c8-4aec-ab43-749702f26fed-client-ca-bundle\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.825573 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.825548    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj75p\" (UniqueName: \"kubernetes.io/projected/f1c00182-20c8-4aec-ab43-749702f26fed-kube-api-access-qj75p\") pod \"metrics-server-5f7d695f6f-5txkx\" (UID: \"f1c00182-20c8-4aec-ab43-749702f26fed\") " pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.945922 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.945827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:12.995568 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.995532    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"]
Jul 07 02:19:12.998048 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:12.998027    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.000793 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.000766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 02:19:13.000793 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.000782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pfkcc\""
Jul 07 02:19:13.007273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.007235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"]
Jul 07 02:19:13.019275 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.019238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2sg6\" (UniqueName: \"kubernetes.io/projected/0cca002c-ca33-4534-96a1-418df6fa1b13-kube-api-access-c2sg6\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.019450 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.019425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cca002c-ca33-4534-96a1-418df6fa1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.120338 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.120294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cca002c-ca33-4534-96a1-418df6fa1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.120519 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.120386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2sg6\" (UniqueName: \"kubernetes.io/projected/0cca002c-ca33-4534-96a1-418df6fa1b13-kube-api-access-c2sg6\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.123350 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.123322    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cca002c-ca33-4534-96a1-418df6fa1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.132164 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.132133    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2sg6\" (UniqueName: \"kubernetes.io/projected/0cca002c-ca33-4534-96a1-418df6fa1b13-kube-api-access-c2sg6\") pod \"monitoring-plugin-d5dcff7cf-8qmf5\" (UID: \"0cca002c-ca33-4534-96a1-418df6fa1b13\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.310040 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.309941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:13.475257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.475223    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-694b9757b-864c7"]
Jul 07 02:19:13.479208 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.479187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.481968 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.481940    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 02:19:13.481968 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.481954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 02:19:13.481968 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.481963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c2pcw\""
Jul 07 02:19:13.482208 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.481974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 02:19:13.482208 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.481943    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 02:19:13.482208 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.482067    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 02:19:13.485677 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:13.485650    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf87aa6_80d0_4e04_b674_3345f5a61a64.slice/crio-72560960f2d7a412303cd3915e0eae20320f9c191af725c1c8c2d7c4b65e139c WatchSource:0}: Error finding container 72560960f2d7a412303cd3915e0eae20320f9c191af725c1c8c2d7c4b65e139c: Status 404 returned error can't find the container with id 72560960f2d7a412303cd3915e0eae20320f9c191af725c1c8c2d7c4b65e139c
Jul 07 02:19:13.488718 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.488525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 02:19:13.492234 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.492211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-694b9757b-864c7"]
Jul 07 02:19:13.525109 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-federate-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525308 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525308 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525308 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-metrics-client-ca\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525466 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-serving-certs-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525466 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525466 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gv4f\" (UniqueName: \"kubernetes.io/projected/e2a84723-1983-477b-8546-fe81d2fc88fd-kube-api-access-9gv4f\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.525604 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.525505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626098 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-metrics-client-ca\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-serving-certs-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv4f\" (UniqueName: \"kubernetes.io/projected/e2a84723-1983-477b-8546-fe81d2fc88fd-kube-api-access-9gv4f\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-federate-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.626673 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.626370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.627078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.627000    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-metrics-client-ca\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.627078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.627007    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-serving-certs-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.627280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.627244    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.629470 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.629444    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-telemeter-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.630020 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.629966    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-federate-client-tls\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.630020 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.629968    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.630144 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.630059    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2a84723-1983-477b-8546-fe81d2fc88fd-secret-telemeter-client\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.635686 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.635658    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv4f\" (UniqueName: \"kubernetes.io/projected/e2a84723-1983-477b-8546-fe81d2fc88fd-kube-api-access-9gv4f\") pod \"telemeter-client-694b9757b-864c7\" (UID: \"e2a84723-1983-477b-8546-fe81d2fc88fd\") " pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:13.732889 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.732853    2561 generic.go:356] "Generic (PLEG): container finished" podID="2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a" containerID="4a9b80a4f4549cd6ec45e5682aa16b86f73eda7063cb7743661e65fb6f2d8cdc" exitCode=0
Jul 07 02:19:13.733053 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.732931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqpwp" event={"ID":"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a","Type":"ContainerDied","Data":"4a9b80a4f4549cd6ec45e5682aa16b86f73eda7063cb7743661e65fb6f2d8cdc"}
Jul 07 02:19:13.734195 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.734173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"72560960f2d7a412303cd3915e0eae20320f9c191af725c1c8c2d7c4b65e139c"}
Jul 07 02:19:13.790663 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:13.790609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-694b9757b-864c7"
Jul 07 02:19:14.296005 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.295920    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:14.326012 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.325980    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:19:14.428541 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.428487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:19:14.428541 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.428534    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:19:14.428823 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.428644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.477332 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.477294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:19:14.477533 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.477492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.480423 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.480398    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akcdlv3424s7u\""
Jul 07 02:19:14.480573 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.480526    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 02:19:14.481424 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 02:19:14.481424 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481414    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 02:19:14.481672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 02:19:14.481672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 02:19:14.481672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 02:19:14.481672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481655    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 02:19:14.481868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481722    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 02:19:14.481868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 02:19:14.481987 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.481971    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 02:19:14.482183 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.482051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 02:19:14.482288 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.482230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 02:19:14.482479 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.482457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cd42j\""
Jul 07 02:19:14.482591 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.482574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 02:19:14.483520 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.483499    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 02:19:14.533940 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.533909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.533940 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.533948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.533971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482gb\" (UniqueName: \"kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwt6v\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.534957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.534924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.635946 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.635914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.635946 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.635951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.635980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.635998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482gb\" (UniqueName: \"kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.636438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwt6v\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636501    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.636993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.637030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.637075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.637099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.637963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.637454    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.638493 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.638465    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.639387 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.639360    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.640429 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.640358    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.640658    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.640879    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.640879    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.640933    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.641420    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.641725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.641449    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642084 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.641732    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642084 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.641935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642084 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.641994    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642234 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.642103    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642994 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.642458    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642994 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.642509    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.642994 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.642649    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.642994 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.642954    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.643218 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.643031    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.643342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.643322    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.643684 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.643662    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.644040 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.644024    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.645229 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.645208    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.645318 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.645284    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482gb\" (UniqueName: \"kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.645388 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.645368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config\") pod \"console-99bc99bc8-wrf9c\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") " pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.649391 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.649364    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwt6v\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v\") pod \"prometheus-k8s-0\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.742060 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.742027    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:14.789189 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.789145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:14.940387 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.940291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:19:14.940387 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.940357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:14.943584 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.943564    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:19:14.943670 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.943564    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:19:14.953127 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.953103    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb558531-20cf-4b45-a955-5f1ca96e1516-original-pull-secret\") pod \"global-pull-secret-syncer-dqm7c\" (UID: \"fb558531-20cf-4b45-a955-5f1ca96e1516\") " pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:19:14.953359 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.953342    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:19:14.963814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.963787    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92t9\" (UniqueName: \"kubernetes.io/projected/d1ddd8ce-320b-4115-a3b5-d7082fe3da1a-kube-api-access-t92t9\") pod \"network-check-target-r2x82\" (UID: \"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a\") " pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:14.993926 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:14.993884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqm7c"
Jul 07 02:19:15.041738 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.041706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:19:15.044377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.044357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:19:15.054605 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.054576    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed4f47c0-1a2e-4045-826a-65ba949859d3-metrics-certs\") pod \"network-metrics-daemon-t5bd7\" (UID: \"ed4f47c0-1a2e-4045-826a-65ba949859d3\") " pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:19:15.097426 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.097398    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv29t\""
Jul 07 02:19:15.105072 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.105041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:15.290897 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.290817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kthkt\""
Jul 07 02:19:15.299114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.299075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5bd7"
Jul 07 02:19:15.747907 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.747844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqpwp" event={"ID":"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a","Type":"ContainerStarted","Data":"094db92bb1e87b8d45de9af3e26a8219ebec2a78bb05b3e8043e74d1474fcecd"}
Jul 07 02:19:15.941488 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:15.941143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r2x82"]
Jul 07 02:19:16.004726 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.004405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f7d695f6f-5txkx"]
Jul 07 02:19:16.024863 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.010637    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-694b9757b-864c7"]
Jul 07 02:19:16.024863 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.010802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"]
Jul 07 02:19:16.024863 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:16.011472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ddd8ce_320b_4115_a3b5_d7082fe3da1a.slice/crio-2c719ed693b73596aeadccdffb0a78533a47c566076b371f964562b518fc09a0 WatchSource:0}: Error finding container 2c719ed693b73596aeadccdffb0a78533a47c566076b371f964562b518fc09a0: Status 404 returned error can't find the container with id 2c719ed693b73596aeadccdffb0a78533a47c566076b371f964562b518fc09a0
Jul 07 02:19:16.037323 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.030915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqm7c"]
Jul 07 02:19:16.057295 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:16.053368    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb558531_20cf_4b45_a955_5f1ca96e1516.slice/crio-59ed3848265894312f437dfb9ae53cbfb9b65d91e00b3bc7858d2290ef076354 WatchSource:0}: Error finding container 59ed3848265894312f437dfb9ae53cbfb9b65d91e00b3bc7858d2290ef076354: Status 404 returned error can't find the container with id 59ed3848265894312f437dfb9ae53cbfb9b65d91e00b3bc7858d2290ef076354
Jul 07 02:19:16.072912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.072057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:19:16.076118 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:16.075894    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1961650c_0655_4f79_b8e5_5a288796228d.slice/crio-586834f0c8d404f7fba202f14bfd09e32d12ae915a5439df6990b29e7e23e62f WatchSource:0}: Error finding container 586834f0c8d404f7fba202f14bfd09e32d12ae915a5439df6990b29e7e23e62f: Status 404 returned error can't find the container with id 586834f0c8d404f7fba202f14bfd09e32d12ae915a5439df6990b29e7e23e62f
Jul 07 02:19:16.085410 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.085379    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:19:16.087261 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.087239    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5bd7"]
Jul 07 02:19:16.099121 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:19:16.098717    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4b9ce3_0377_4dbd_afea_4a510f6df0e8.slice/crio-87f463eee0167218c8cd4c0b3ff0358347b862542fa24e58d50f5a09178ecf04 WatchSource:0}: Error finding container 87f463eee0167218c8cd4c0b3ff0358347b862542fa24e58d50f5a09178ecf04: Status 404 returned error can't find the container with id 87f463eee0167218c8cd4c0b3ff0358347b862542fa24e58d50f5a09178ecf04
Jul 07 02:19:16.759005 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.758959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqm7c" event={"ID":"fb558531-20cf-4b45-a955-5f1ca96e1516","Type":"ContainerStarted","Data":"59ed3848265894312f437dfb9ae53cbfb9b65d91e00b3bc7858d2290ef076354"}
Jul 07 02:19:16.760992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.760943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5bd7" event={"ID":"ed4f47c0-1a2e-4045-826a-65ba949859d3","Type":"ContainerStarted","Data":"d2fdf192669ba4534d091699a3af1ae08d7ae74dd58edab3fb22f5d0bc18bb88"}
Jul 07 02:19:16.765220 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.765115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99bc99bc8-wrf9c" event={"ID":"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8","Type":"ContainerStarted","Data":"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"}
Jul 07 02:19:16.765220 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.765166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99bc99bc8-wrf9c" event={"ID":"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8","Type":"ContainerStarted","Data":"87f463eee0167218c8cd4c0b3ff0358347b862542fa24e58d50f5a09178ecf04"}
Jul 07 02:19:16.770404 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.769455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9c69899-dpl7z" event={"ID":"1178fbe8-d6ea-42c6-a885-3d282a608cde","Type":"ContainerStarted","Data":"7564e5099cbf8843b43b7e9cb6d557c0544a70da72c4b228ee7e515d8939d379"}
Jul 07 02:19:16.779590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.778874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk" event={"ID":"8312a6ec-7348-4770-baec-9bab8b97eb2f","Type":"ContainerStarted","Data":"f32b7793998e7223b15c70a70cec375db465c057a75e69a54397c98291cbf37c"}
Jul 07 02:19:16.783889 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.783831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r2x82" event={"ID":"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a","Type":"ContainerStarted","Data":"2c719ed693b73596aeadccdffb0a78533a47c566076b371f964562b518fc09a0"}
Jul 07 02:19:16.788043 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.787986    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751" exitCode=0
Jul 07 02:19:16.788173 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.788061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"}
Jul 07 02:19:16.788173 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.788124    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"586834f0c8d404f7fba202f14bfd09e32d12ae915a5439df6990b29e7e23e62f"}
Jul 07 02:19:16.793357 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.793257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5" event={"ID":"0cca002c-ca33-4534-96a1-418df6fa1b13","Type":"ContainerStarted","Data":"80669e500b6176afdf674d01816d12b9cccc0a5efd6e3a9588bc0f5c074fb6f6"}
Jul 07 02:19:16.799006 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.798301    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d" exitCode=0
Jul 07 02:19:16.799006 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.798384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"}
Jul 07 02:19:16.802064 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.801925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694b9757b-864c7" event={"ID":"e2a84723-1983-477b-8546-fe81d2fc88fd","Type":"ContainerStarted","Data":"2df5c01d183c811561473a69a7ae2e9c951d4f892fb2dc48511e69bdac3447c3"}
Jul 07 02:19:16.805079 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.804905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx" event={"ID":"f1c00182-20c8-4aec-ab43-749702f26fed","Type":"ContainerStarted","Data":"ca7c151b41d6b4f2de7c8b6bbd814c16c4eedf4834b5c5d43201624713e3c296"}
Jul 07 02:19:16.806020 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.805973    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d9c69899-dpl7z" podStartSLOduration=2.634416092 podStartE2EDuration="8.805940388s" totalImagesPullingTime="6.171524296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.805275786 +0000 UTC m=+68.112594176" watchObservedRunningTime="2026-07-07 02:19:16.805940388 +0000 UTC m=+68.113258779"
Jul 07 02:19:16.806423 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.806373    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-99bc99bc8-wrf9c" podStartSLOduration=2.8063590290000002 podStartE2EDuration="2.806359029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.786059584 +0000 UTC m=+68.093377974" watchObservedRunningTime="2026-07-07 02:19:16.806359029 +0000 UTC m=+68.113677421"
Jul 07 02:19:16.810212 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.810182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqpwp" event={"ID":"2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a","Type":"ContainerStarted","Data":"5299f0c0cf9782728f6d7055b3298b0e949c2f75abb10370e973aa3232b32897"}
Jul 07 02:19:16.824644 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.823877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx" event={"ID":"b8789e80-7a85-4faf-be6a-12a4bfeb2491","Type":"ContainerStarted","Data":"9dc50c07e6a2d4eee305fb619c8ea35e98176224cc1eadabc4d654e3f0343719"}
Jul 07 02:19:16.824644 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.823943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx" event={"ID":"b8789e80-7a85-4faf-be6a-12a4bfeb2491","Type":"ContainerStarted","Data":"8b0e8f02660ac8f389eee9a1bffe7381ec8ce4d71f856ce543183e84554f5369"}
Jul 07 02:19:16.824644 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.823975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx" event={"ID":"b8789e80-7a85-4faf-be6a-12a4bfeb2491","Type":"ContainerStarted","Data":"f304c627acc34ccdac4ab3571436da64e1aedcc305c20e226755d1d0b8027e90"}
Jul 07 02:19:16.828576 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.828517    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z8nbk" podStartSLOduration=3.657732478 podStartE2EDuration="8.828494302s" totalImagesPullingTime="5.170761824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.825406478 +0000 UTC m=+68.132724869" watchObservedRunningTime="2026-07-07 02:19:16.828494302 +0000 UTC m=+68.135812718"
Jul 07 02:19:16.833810 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.832393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlvjt" event={"ID":"0823e9dd-da0a-4311-b643-5519fcf57046","Type":"ContainerStarted","Data":"2191b1bf1e9688802d6f9ff070141ae9ff5e4fc30934b9afa828524e86a8709d"}
Jul 07 02:19:16.869353 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.869084    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dmsjx" podStartSLOduration=3.55819454 podStartE2EDuration="8.869058705s" totalImagesPullingTime="5.310864165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.867695187 +0000 UTC m=+68.175013578" watchObservedRunningTime="2026-07-07 02:19:16.869058705 +0000 UTC m=+68.176377096"
Jul 07 02:19:16.923896 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:16.923668    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dqpwp" podStartSLOduration=7.38122053 podStartE2EDuration="8.923644717s" totalImagesPullingTime="1.41995398s" totalInitContainerRuntime="122.470207ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.921389799 +0000 UTC m=+68.228708208" watchObservedRunningTime="2026-07-07 02:19:16.923644717 +0000 UTC m=+68.230963099"
Jul 07 02:19:19.210559 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:19.210528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:19.717930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:19.717903    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jtz8x"
Jul 07 02:19:19.736934 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:19.736876    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dlvjt" podStartSLOduration=5.844325362 podStartE2EDuration="17.736857561s" totalImagesPullingTime="11.892532199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:19:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.942265445 +0000 UTC m=+68.249583837" watchObservedRunningTime="2026-07-07 02:19:19.736857561 +0000 UTC m=+71.044175950"
Jul 07 02:19:24.742155 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.742125    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:24.742761 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.742225    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:24.747532 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.747508    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:24.876782 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.876739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"027e83b322dd9a991f3b05a6f1c6a9855450c6e26bc9d61d4c6f53f5f89a3db8"}
Jul 07 02:19:24.876992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.876838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"b1e953e85f3bde9561abb73af62472240a43665ef9f946cb9f35b33c1bd11b27"}
Jul 07 02:19:24.876992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.876874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"9163651bc09820dcaeb18f6be07b51b63f6e90cd56d1bfbabbfba678b07d29fa"}
Jul 07 02:19:24.878699 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.878662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqm7c" event={"ID":"fb558531-20cf-4b45-a955-5f1ca96e1516","Type":"ContainerStarted","Data":"ebfe6d49104163b0246f9688eb38c79fcb04a5f1f9667766091f475c5f3617e1"}
Jul 07 02:19:24.881015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.880972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5bd7" event={"ID":"ed4f47c0-1a2e-4045-826a-65ba949859d3","Type":"ContainerStarted","Data":"33b52d00a30e52e06b9e7113b0ca2fea049090c6419c2df829bc551b27ced443"}
Jul 07 02:19:24.881134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.881049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5bd7" event={"ID":"ed4f47c0-1a2e-4045-826a-65ba949859d3","Type":"ContainerStarted","Data":"8f5b6985a2aefe855535cabeec80b9f592a7956fa89ed5633e6948230043f2ad"}
Jul 07 02:19:24.882791 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.882767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r2x82" event={"ID":"d1ddd8ce-320b-4115-a3b5-d7082fe3da1a","Type":"ContainerStarted","Data":"a24f425a63d0629eaf4015f1663ba9e688db82b7e283ea76920a0cf0fcdf692a"}
Jul 07 02:19:24.882932 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.882833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:19:24.886427 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"}
Jul 07 02:19:24.886538 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"}
Jul 07 02:19:24.886538 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886522    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"}
Jul 07 02:19:24.886669 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"}
Jul 07 02:19:24.886669 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"}
Jul 07 02:19:24.886777 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.886673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerStarted","Data":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"}
Jul 07 02:19:24.887980 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.887959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5" event={"ID":"0cca002c-ca33-4534-96a1-418df6fa1b13","Type":"ContainerStarted","Data":"b968dd9819026820876ffeb70a381c684a191ae2f74a4de32bc1e4b3a0e436f5"}
Jul 07 02:19:24.888113 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.888095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:24.891860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.891825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"}
Jul 07 02:19:24.892041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.892015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"}
Jul 07 02:19:24.892176 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.892061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"}
Jul 07 02:19:24.892176 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.892087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"}
Jul 07 02:19:24.892176 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.892126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"}
Jul 07 02:19:24.895103 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.895078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5"
Jul 07 02:19:24.895841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.895802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dqm7c" podStartSLOduration=68.000220273 podStartE2EDuration="1m15.895763019s" totalImagesPullingTime="7.895542746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.894303025 +0000 UTC m=+76.201621416" watchObservedRunningTime="2026-07-07 02:19:24.895763019 +0000 UTC m=+76.203081409"
Jul 07 02:19:24.896375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.896353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694b9757b-864c7" event={"ID":"e2a84723-1983-477b-8546-fe81d2fc88fd","Type":"ContainerStarted","Data":"0a760dcee40dd8ccd091f103151aca3bbb5925692ed84d4226a5a8d83a883c6b"}
Jul 07 02:19:24.896495 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.896402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694b9757b-864c7" event={"ID":"e2a84723-1983-477b-8546-fe81d2fc88fd","Type":"ContainerStarted","Data":"5653fb2ebd1c20be9c2ecc57ff421bfd1d327569f037a72aac95eedd490ca576"}
Jul 07 02:19:24.896495 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.896427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694b9757b-864c7" event={"ID":"e2a84723-1983-477b-8546-fe81d2fc88fd","Type":"ContainerStarted","Data":"7520682c60cad18a93782e72f389ac357575797bac937c8740dc2664ad3721bf"}
Jul 07 02:19:24.898037 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.898012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx" event={"ID":"f1c00182-20c8-4aec-ab43-749702f26fed","Type":"ContainerStarted","Data":"2ca3a013dda753060290afb02e48367ed72253d3fe2a404e586f066281b27fd4"}
Jul 07 02:19:24.903571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.903546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:19:24.912382 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.912280    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t5bd7" podStartSLOduration=68.060187073 podStartE2EDuration="1m15.912258673s" totalImagesPullingTime="7.8520716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.911272686 +0000 UTC m=+76.218591083" watchObservedRunningTime="2026-07-07 02:19:24.912258673 +0000 UTC m=+76.219577063"
Jul 07 02:19:24.927777 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.927723    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8qmf5" podStartSLOduration=5.0291956 podStartE2EDuration="12.927708146s" totalImagesPullingTime="7.898512546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.926106682 +0000 UTC m=+76.233425071" watchObservedRunningTime="2026-07-07 02:19:24.927708146 +0000 UTC m=+76.235026537"
Jul 07 02:19:24.944714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.944617    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r2x82" podStartSLOduration=68.007034314 podStartE2EDuration="1m15.94459344s" totalImagesPullingTime="7.937559126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.94328969 +0000 UTC m=+76.250608081" watchObservedRunningTime="2026-07-07 02:19:24.94459344 +0000 UTC m=+76.251911830"
Jul 07 02:19:24.975939 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.975883    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.768070406 podStartE2EDuration="10.975861341s" totalImagesPullingTime="7.179776878s" totalInitContainerRuntime="28.014057ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.972980132 +0000 UTC m=+76.280298523" watchObservedRunningTime="2026-07-07 02:19:24.975861341 +0000 UTC m=+76.283179731"
Jul 07 02:19:24.994993 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:24.994929    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx" podStartSLOduration=5.139696341 podStartE2EDuration="12.994906498s" totalImagesPullingTime="7.855210157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:24.992599334 +0000 UTC m=+76.299917719" watchObservedRunningTime="2026-07-07 02:19:24.994906498 +0000 UTC m=+76.302224888"
Jul 07 02:19:25.022863 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.021383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-694b9757b-864c7" podStartSLOduration=4.11960366 podStartE2EDuration="12.021362221s" totalImagesPullingTime="7.901758561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:25.017373533 +0000 UTC m=+76.324691923" watchObservedRunningTime="2026-07-07 02:19:25.021362221 +0000 UTC m=+76.328680612"
Jul 07 02:19:25.905916 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.905878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerStarted","Data":"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"}
Jul 07 02:19:25.909310 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.909245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"436aea444ca57f9064b9e0f3b83fd7d7608356850ac29b559bc5bb130f1dfa0f"}
Jul 07 02:19:25.909441 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.909319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"6a3b1fd52f2a90ebc2f506b682608660ec93860630d8f7bc435fbbc85c88ac05"}
Jul 07 02:19:25.909441 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.909363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" event={"ID":"dbf87aa6-80d0-4e04-b674-3345f5a61a64","Type":"ContainerStarted","Data":"5783e0394e16af12d80eff4609904ca76b7fac4732f72fa9c22b1ed67822c69b"}
Jul 07 02:19:25.940267 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.940216    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.964916462 podStartE2EDuration="16.940199414s" totalImagesPullingTime="12.975282952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:25.937705055 +0000 UTC m=+77.245023446" watchObservedRunningTime="2026-07-07 02:19:25.940199414 +0000 UTC m=+77.247517805"
Jul 07 02:19:25.967287 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:25.967224    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87" podStartSLOduration=4.728548901 podStartE2EDuration="14.967199771s" totalImagesPullingTime="10.23865087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:25.965569213 +0000 UTC m=+77.272887628" watchObservedRunningTime="2026-07-07 02:19:25.967199771 +0000 UTC m=+77.274518163"
Jul 07 02:19:26.914664 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:26.914605    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:29.789404 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:29.789253    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:19:32.925338 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:32.925311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-66d5b8bbf6-dfk87"
Jul 07 02:19:32.946221 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:32.946192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:32.946382 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:32.946242    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:37.087822 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:37.087786    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:19:41.835482 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:41.835435    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8d9c69899-dpl7z" podUID="1178fbe8-d6ea-42c6-a885-3d282a608cde" containerName="console" containerID="cri-o://7564e5099cbf8843b43b7e9cb6d557c0544a70da72c4b228ee7e515d8939d379" gracePeriod=15
Jul 07 02:19:41.984063 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:41.984026    2561 generic.go:356] "Generic (PLEG): container finished" podID="1178fbe8-d6ea-42c6-a885-3d282a608cde" containerID="7564e5099cbf8843b43b7e9cb6d557c0544a70da72c4b228ee7e515d8939d379" exitCode=0
Jul 07 02:19:41.984233 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:41.984095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9c69899-dpl7z" event={"ID":"1178fbe8-d6ea-42c6-a885-3d282a608cde","Type":"ContainerDied","Data":"7564e5099cbf8843b43b7e9cb6d557c0544a70da72c4b228ee7e515d8939d379"}
Jul 07 02:19:42.082309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.082280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:42.220527 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220527 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpjpb\" (UniqueName: \"kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220527 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220770    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle\") pod \"1178fbe8-d6ea-42c6-a885-3d282a608cde\" (UID: \"1178fbe8-d6ea-42c6-a885-3d282a608cde\") "
Jul 07 02:19:42.220969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220917    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca" (OuterVolumeSpecName: "service-ca") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:19:42.220969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.220921    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config" (OuterVolumeSpecName: "console-config") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:19:42.221133 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.221122    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.221190 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.221141    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-service-ca\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.221239 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.221191    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:19:42.221343 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.221315    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:19:42.223251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.223215    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:19:42.223251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.223223    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb" (OuterVolumeSpecName: "kube-api-access-dpjpb") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "kube-api-access-dpjpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:19:42.223378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.223261    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1178fbe8-d6ea-42c6-a885-3d282a608cde" (UID: "1178fbe8-d6ea-42c6-a885-3d282a608cde"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:19:42.322243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.322187    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-serving-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.322243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.322235    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-oauth-serving-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.322243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.322245    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1178fbe8-d6ea-42c6-a885-3d282a608cde-console-oauth-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.322243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.322254    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1178fbe8-d6ea-42c6-a885-3d282a608cde-trusted-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.322243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.322263    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjpb\" (UniqueName: \"kubernetes.io/projected/1178fbe8-d6ea-42c6-a885-3d282a608cde-kube-api-access-dpjpb\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:19:42.989850 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.989821    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9c69899-dpl7z"
Jul 07 02:19:42.990294 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.989820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9c69899-dpl7z" event={"ID":"1178fbe8-d6ea-42c6-a885-3d282a608cde","Type":"ContainerDied","Data":"c6a6a5fddf5034717289d55071f5fa44a1a50861f5f068d28c852601ab55c95b"}
Jul 07 02:19:42.990294 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:42.989934    2561 scope.go:117] "RemoveContainer" containerID="7564e5099cbf8843b43b7e9cb6d557c0544a70da72c4b228ee7e515d8939d379"
Jul 07 02:19:43.012283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:43.012253    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:43.019211 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:43.019183    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d9c69899-dpl7z"]
Jul 07 02:19:43.286069 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:43.285981    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1178fbe8-d6ea-42c6-a885-3d282a608cde" path="/var/lib/kubelet/pods/1178fbe8-d6ea-42c6-a885-3d282a608cde/volumes"
Jul 07 02:19:51.608347 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:51.608315    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rlhqp_ec88dc48-87af-434b-b851-19eb2e8d449f/serve-healthcheck-canary/0.log"
Jul 07 02:19:52.953483 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:52.953445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:52.957578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:52.957549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f7d695f6f-5txkx"
Jul 07 02:19:55.911917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:19:55.911886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r2x82"
Jul 07 02:20:02.111613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.111567    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-99bc99bc8-wrf9c" podUID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" containerName="console" containerID="cri-o://a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8" gracePeriod=15
Jul 07 02:20:02.374639 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.374544    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:20:02.475557 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.475764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.475764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-482gb\" (UniqueName: \"kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.475764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.475764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.475977 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.475953    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:02.476167 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476132    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config" (OuterVolumeSpecName: "console-config") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:02.476167 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476138    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:02.476525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.476525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert\") pod \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\" (UID: \"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8\") "
Jul 07 02:20:02.476661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476567    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-service-ca\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.476661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476570    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:02.476661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476586    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.476661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.476601    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-trusted-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.478177 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.478141    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb" (OuterVolumeSpecName: "kube-api-access-482gb") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "kube-api-access-482gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:20:02.478177 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.478156    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:02.478484 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.478461    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" (UID: "ac4b9ce3-0377-4dbd-afea-4a510f6df0e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:02.577858 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.577823    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-oauth-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.577858 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.577856    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482gb\" (UniqueName: \"kubernetes.io/projected/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-kube-api-access-482gb\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.577858 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.577870    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-oauth-serving-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:02.578095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:02.577882    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8-console-serving-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:03.076507 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.076474    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" containerID="a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8" exitCode=0
Jul 07 02:20:03.076737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.076523    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99bc99bc8-wrf9c"
Jul 07 02:20:03.076737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.076543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99bc99bc8-wrf9c" event={"ID":"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8","Type":"ContainerDied","Data":"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"}
Jul 07 02:20:03.076737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.076612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99bc99bc8-wrf9c" event={"ID":"ac4b9ce3-0377-4dbd-afea-4a510f6df0e8","Type":"ContainerDied","Data":"87f463eee0167218c8cd4c0b3ff0358347b862542fa24e58d50f5a09178ecf04"}
Jul 07 02:20:03.076737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.076665    2561 scope.go:117] "RemoveContainer" containerID="a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"
Jul 07 02:20:03.086002 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.085983    2561 scope.go:117] "RemoveContainer" containerID="a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"
Jul 07 02:20:03.086296 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:03.086274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8\": container with ID starting with a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8 not found: ID does not exist" containerID="a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"
Jul 07 02:20:03.086351 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.086306    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8"} err="failed to get container status \"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8\": rpc error: code = NotFound desc = could not find container \"a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8\": container with ID starting with a5f87d8c34a0dd9daff0f236f8d1f0770594643103621c5c75fc56a003a5bef8 not found: ID does not exist"
Jul 07 02:20:03.101076 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.101044    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:20:03.105657 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.105608    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-99bc99bc8-wrf9c"]
Jul 07 02:20:03.285686 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:03.285650    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" path="/var/lib/kubelet/pods/ac4b9ce3-0377-4dbd-afea-4a510f6df0e8/volumes"
Jul 07 02:20:14.789561 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:14.789500    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:14.810524 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:14.810499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:15.144397 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:15.144368    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:28.567845 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.567806    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:28.568280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568086    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="alertmanager" containerID="cri-o://02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08" gracePeriod=120
Jul 07 02:20:28.568280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568186    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de" gracePeriod=120
Jul 07 02:20:28.568280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568185    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy" containerID="cri-o://3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a" gracePeriod=120
Jul 07 02:20:28.568280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568222    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="prom-label-proxy" containerID="cri-o://f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43" gracePeriod=120
Jul 07 02:20:28.568280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568238    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="config-reloader" containerID="cri-o://fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f" gracePeriod=120
Jul 07 02:20:28.568534 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:28.568192    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-web" containerID="cri-o://5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf" gracePeriod=120
Jul 07 02:20:29.187034 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.186999    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43" exitCode=0
Jul 07 02:20:29.187034 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187026    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a" exitCode=0
Jul 07 02:20:29.187034 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187032    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f" exitCode=0
Jul 07 02:20:29.187034 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187038    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08" exitCode=0
Jul 07 02:20:29.187297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"}
Jul 07 02:20:29.187297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"}
Jul 07 02:20:29.187297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"}
Jul 07 02:20:29.187297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.187236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"}
Jul 07 02:20:29.824274 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.824251    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:29.922704 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.922704 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.922704 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922692    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.922973 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfxxn\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.922973 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.922973 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.922974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923054    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923397 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923397 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") pod \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\" (UID: \"1160d5d2-4d22-496a-bcb3-3b23bff0f402\") "
Jul 07 02:20:29.923911 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.923742    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:29.924043 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.924008    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:29.924689 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.924370    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:20:29.926363 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.926308    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.926466 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.926363    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn" (OuterVolumeSpecName: "kube-api-access-sfxxn") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "kube-api-access-sfxxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:20:29.926535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.926460    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume" (OuterVolumeSpecName: "config-volume") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.926535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.926469    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out" (OuterVolumeSpecName: "config-out") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:20:29.926689 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.926563    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:20:29.927060 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.927032    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.928271 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.928251    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.928784 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.928765    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.930903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.930872    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:29.938009 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:29.937987    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config" (OuterVolumeSpecName: "web-config") pod "1160d5d2-4d22-496a-bcb3-3b23bff0f402" (UID: "1160d5d2-4d22-496a-bcb3-3b23bff0f402"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:30.024860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024823    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-web-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.024860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024855    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.024860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024866    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024876    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024886    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024896    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-volume\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024904    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-tls-assets\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024911    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-cluster-tls-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024919    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxxn\" (UniqueName: \"kubernetes.io/projected/1160d5d2-4d22-496a-bcb3-3b23bff0f402-kube-api-access-sfxxn\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024927    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-config-out\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024934    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1160d5d2-4d22-496a-bcb3-3b23bff0f402-metrics-client-ca\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024942    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1160d5d2-4d22-496a-bcb3-3b23bff0f402-alertmanager-main-db\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.025090 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.024950    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1160d5d2-4d22-496a-bcb3-3b23bff0f402-secret-alertmanager-main-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:30.193911 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.193827    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de" exitCode=0
Jul 07 02:20:30.193911 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.193853    2561 generic.go:356] "Generic (PLEG): container finished" podID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerID="5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf" exitCode=0
Jul 07 02:20:30.194086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.193914    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.194086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.193913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"}
Jul 07 02:20:30.194086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.193984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"}
Jul 07 02:20:30.194086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.194014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1160d5d2-4d22-496a-bcb3-3b23bff0f402","Type":"ContainerDied","Data":"3b964eb8192cec791932f4bd69fc064510ac3532f7251ea9e645d65d74732294"}
Jul 07 02:20:30.194086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.194043    2561 scope.go:117] "RemoveContainer" containerID="f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"
Jul 07 02:20:30.202305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.202288    2561 scope.go:117] "RemoveContainer" containerID="8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"
Jul 07 02:20:30.209461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.209440    2561 scope.go:117] "RemoveContainer" containerID="3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"
Jul 07 02:20:30.216781 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.216762    2561 scope.go:117] "RemoveContainer" containerID="5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"
Jul 07 02:20:30.219786 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.219715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:30.225356 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.225336    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:30.225432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.225371    2561 scope.go:117] "RemoveContainer" containerID="fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"
Jul 07 02:20:30.232599 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.232579    2561 scope.go:117] "RemoveContainer" containerID="02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"
Jul 07 02:20:30.239764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.239745    2561 scope.go:117] "RemoveContainer" containerID="82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"
Jul 07 02:20:30.247485 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.247467    2561 scope.go:117] "RemoveContainer" containerID="f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"
Jul 07 02:20:30.247833 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.247815    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43\": container with ID starting with f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43 not found: ID does not exist" containerID="f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"
Jul 07 02:20:30.247883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.247841    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"} err="failed to get container status \"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43\": rpc error: code = NotFound desc = could not find container \"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43\": container with ID starting with f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43 not found: ID does not exist"
Jul 07 02:20:30.247883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.247861    2561 scope.go:117] "RemoveContainer" containerID="8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"
Jul 07 02:20:30.248078 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.248065    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de\": container with ID starting with 8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de not found: ID does not exist" containerID="8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"
Jul 07 02:20:30.248117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248080    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"} err="failed to get container status \"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de\": rpc error: code = NotFound desc = could not find container \"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de\": container with ID starting with 8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de not found: ID does not exist"
Jul 07 02:20:30.248117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248092    2561 scope.go:117] "RemoveContainer" containerID="3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"
Jul 07 02:20:30.248295 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.248281    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a\": container with ID starting with 3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a not found: ID does not exist" containerID="3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"
Jul 07 02:20:30.248339 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"} err="failed to get container status \"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a\": rpc error: code = NotFound desc = could not find container \"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a\": container with ID starting with 3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a not found: ID does not exist"
Jul 07 02:20:30.248339 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248315    2561 scope.go:117] "RemoveContainer" containerID="5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"
Jul 07 02:20:30.248538 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.248520    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf\": container with ID starting with 5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf not found: ID does not exist" containerID="5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"
Jul 07 02:20:30.248601 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248548    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"} err="failed to get container status \"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf\": rpc error: code = NotFound desc = could not find container \"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf\": container with ID starting with 5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf not found: ID does not exist"
Jul 07 02:20:30.248601 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248573    2561 scope.go:117] "RemoveContainer" containerID="fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"
Jul 07 02:20:30.248813 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.248793    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f\": container with ID starting with fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f not found: ID does not exist" containerID="fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"
Jul 07 02:20:30.248854 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248819    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"} err="failed to get container status \"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f\": rpc error: code = NotFound desc = could not find container \"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f\": container with ID starting with fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f not found: ID does not exist"
Jul 07 02:20:30.248854 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.248833    2561 scope.go:117] "RemoveContainer" containerID="02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"
Jul 07 02:20:30.249055 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.249040    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08\": container with ID starting with 02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08 not found: ID does not exist" containerID="02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"
Jul 07 02:20:30.249094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249058    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"} err="failed to get container status \"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08\": rpc error: code = NotFound desc = could not find container \"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08\": container with ID starting with 02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08 not found: ID does not exist"
Jul 07 02:20:30.249094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249071    2561 scope.go:117] "RemoveContainer" containerID="82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"
Jul 07 02:20:30.249279 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:30.249264    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d\": container with ID starting with 82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d not found: ID does not exist" containerID="82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"
Jul 07 02:20:30.249316 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249283    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"} err="failed to get container status \"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d\": rpc error: code = NotFound desc = could not find container \"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d\": container with ID starting with 82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d not found: ID does not exist"
Jul 07 02:20:30.249316 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249296    2561 scope.go:117] "RemoveContainer" containerID="f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"
Jul 07 02:20:30.249488 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249473    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43"} err="failed to get container status \"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43\": rpc error: code = NotFound desc = could not find container \"f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43\": container with ID starting with f26c4d79de1cd88acb9601e2927e958544f7955a5234e9e86d3a8142c20b7a43 not found: ID does not exist"
Jul 07 02:20:30.249524 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249489    2561 scope.go:117] "RemoveContainer" containerID="8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"
Jul 07 02:20:30.249721 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249704    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de"} err="failed to get container status \"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de\": rpc error: code = NotFound desc = could not find container \"8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de\": container with ID starting with 8a6b3987726231aefc277e8d9dfb780eb0f9769ed1195c4b7546825d966eb6de not found: ID does not exist"
Jul 07 02:20:30.249773 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249722    2561 scope.go:117] "RemoveContainer" containerID="3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"
Jul 07 02:20:30.249951 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249933    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a"} err="failed to get container status \"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a\": rpc error: code = NotFound desc = could not find container \"3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a\": container with ID starting with 3627ac68c1ce4ec5d97a0205cee8f62850e4295940771bed1434af7f9a12b81a not found: ID does not exist"
Jul 07 02:20:30.250017 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.249953    2561 scope.go:117] "RemoveContainer" containerID="5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"
Jul 07 02:20:30.250171 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250155    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf"} err="failed to get container status \"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf\": rpc error: code = NotFound desc = could not find container \"5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf\": container with ID starting with 5b0a325b44b41763c04414d9634dc49edb7322a5e52ac796d51d1fb8f7fbfecf not found: ID does not exist"
Jul 07 02:20:30.250235 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250173    2561 scope.go:117] "RemoveContainer" containerID="fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"
Jul 07 02:20:30.250380 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250365    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f"} err="failed to get container status \"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f\": rpc error: code = NotFound desc = could not find container \"fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f\": container with ID starting with fc697d917a97746de884445de0b2ffd4f9aef6b730bc76791dada5df0d2b2a1f not found: ID does not exist"
Jul 07 02:20:30.250447 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250381    2561 scope.go:117] "RemoveContainer" containerID="02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"
Jul 07 02:20:30.250577 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250560    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08"} err="failed to get container status \"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08\": rpc error: code = NotFound desc = could not find container \"02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08\": container with ID starting with 02989c4fea961681161034b92b5a64e30c24516217643a2670f44b5b7e850e08 not found: ID does not exist"
Jul 07 02:20:30.250617 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250578    2561 scope.go:117] "RemoveContainer" containerID="82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"
Jul 07 02:20:30.250883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.250850    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d"} err="failed to get container status \"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d\": rpc error: code = NotFound desc = could not find container \"82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d\": container with ID starting with 82c4e69caf2321a6a4e8fc3c0ee319ad9a13a88c7e65d176dc5318d41432954d not found: ID does not exist"
Jul 07 02:20:30.255140 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255120    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:30.255406 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255394    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1178fbe8-d6ea-42c6-a885-3d282a608cde" containerName="console"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255409    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1178fbe8-d6ea-42c6-a885-3d282a608cde" containerName="console"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255419    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="alertmanager"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255424    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="alertmanager"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255433    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-web"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255438    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-web"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255450    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-metric"
Jul 07 02:20:30.255454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255455    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-metric"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255461    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="prom-label-proxy"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255466    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="prom-label-proxy"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255477    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" containerName="console"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255481    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" containerName="console"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255490    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="init-config-reloader"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255495    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="init-config-reloader"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255501    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255515    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="config-reloader"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255520    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="config-reloader"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255562    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4b9ce3-0377-4dbd-afea-4a510f6df0e8" containerName="console"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255569    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="config-reloader"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255575    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-metric"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255581    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy-web"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255589    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1178fbe8-d6ea-42c6-a885-3d282a608cde" containerName="console"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255594    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="alertmanager"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255600    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="prom-label-proxy"
Jul 07 02:20:30.255677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.255606    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" containerName="kube-rbac-proxy"
Jul 07 02:20:30.259309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.259293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.264268 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264251    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 02:20:30.264369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 02:20:30.264516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264499    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 02:20:30.264562 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 02:20:30.264562 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 02:20:30.264851 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.264835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 02:20:30.265059 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.265043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t5lx7\""
Jul 07 02:20:30.265116 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.265079    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 02:20:30.265401 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.265373    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 02:20:30.271742 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.271723    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 02:20:30.278259 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.278235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:30.327377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-volume\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327590 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-out\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327820 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327820 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327820 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jrk\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-kube-api-access-44jrk\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.327820 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.327718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-web-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428491 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-volume\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428491 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-out\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jrk\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-kube-api-access-44jrk\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-web-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.428779 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.429160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.429160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.429160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.428844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.431318 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.430210    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.431318 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.431010    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.431524 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.429249    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.431963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.431932    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432029    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432078 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432043    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-out\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432191 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432102    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432254 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432216    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432244    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.432364 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.432307    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-config-volume\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.433376 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.433355    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-web-config\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.433493 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.433476    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f04d47c4-c2ba-4d44-9be2-2aefb4592436-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.437732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.437714    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jrk\" (UniqueName: \"kubernetes.io/projected/f04d47c4-c2ba-4d44-9be2-2aefb4592436-kube-api-access-44jrk\") pod \"alertmanager-main-0\" (UID: \"f04d47c4-c2ba-4d44-9be2-2aefb4592436\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.568838 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.568809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:20:30.721088 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:30.721046    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:20:30.724320 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:20:30.724293    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04d47c4_c2ba_4d44_9be2_2aefb4592436.slice/crio-41b3636508ac6dfe49a4e167a3b631e6bb3091ef99bf0de870bc3815a4f15740 WatchSource:0}: Error finding container 41b3636508ac6dfe49a4e167a3b631e6bb3091ef99bf0de870bc3815a4f15740: Status 404 returned error can't find the container with id 41b3636508ac6dfe49a4e167a3b631e6bb3091ef99bf0de870bc3815a4f15740
Jul 07 02:20:31.200434 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:31.200397    2561 generic.go:356] "Generic (PLEG): container finished" podID="f04d47c4-c2ba-4d44-9be2-2aefb4592436" containerID="412d4c210a28a790debf39bbee61193f6d3492b42289dd5e4b6dbb4dfea860b6" exitCode=0
Jul 07 02:20:31.200938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:31.200494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerDied","Data":"412d4c210a28a790debf39bbee61193f6d3492b42289dd5e4b6dbb4dfea860b6"}
Jul 07 02:20:31.200938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:31.200577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"41b3636508ac6dfe49a4e167a3b631e6bb3091ef99bf0de870bc3815a4f15740"}
Jul 07 02:20:31.285099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:31.285062    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1160d5d2-4d22-496a-bcb3-3b23bff0f402" path="/var/lib/kubelet/pods/1160d5d2-4d22-496a-bcb3-3b23bff0f402/volumes"
Jul 07 02:20:32.209259 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"61b31619f7fffe1ec221281b8f209818e1574ecf0bc17cb3993d1b3561dcf2dd"}
Jul 07 02:20:32.209668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"88ae4c64328c5cf7bc8277f06d5069db2db3289fde0bac419b5341549dc47d0c"}
Jul 07 02:20:32.209668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"f7933402cf7a04fb5672e98f0d8bb3f934b29bb2cdf3a24e96a5c0db90d678ba"}
Jul 07 02:20:32.209668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"8cee5dfbc7f09e091e349d371931a09b2a3463ee153a6f59b2bc3a41d7ff4a40"}
Jul 07 02:20:32.209668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"54be016328cbb12d7a494e36b07ea35d89aa62b732d865dc2b5ffa8c90e91315"}
Jul 07 02:20:32.209668 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.209397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f04d47c4-c2ba-4d44-9be2-2aefb4592436","Type":"ContainerStarted","Data":"ca7c7f71c676c9e649cc5ad1efdc6a8430008024e99d29e89f445e366b10ca48"}
Jul 07 02:20:32.244806 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.244741    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2289518839999998 podStartE2EDuration="2.244718531s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.766647ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:20:32.24340461 +0000 UTC m=+143.550723013" watchObservedRunningTime="2026-07-07 02:20:32.244718531 +0000 UTC m=+143.552036928"
Jul 07 02:20:32.828697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.828660    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:32.829436 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.829400    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="prometheus" containerID="cri-o://3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6" gracePeriod=600
Jul 07 02:20:32.830410 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.830342    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="thanos-sidecar" containerID="cri-o://83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031" gracePeriod=600
Jul 07 02:20:32.830528 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.830503    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566" gracePeriod=600
Jul 07 02:20:32.830730 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.830525    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy" containerID="cri-o://6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f" gracePeriod=600
Jul 07 02:20:32.830919 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.830550    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-web" containerID="cri-o://cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d" gracePeriod=600
Jul 07 02:20:32.831043 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:32.830570    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="config-reloader" containerID="cri-o://5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995" gracePeriod=600
Jul 07 02:20:33.082966 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.082905    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.152073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152081    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152340 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152111    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152340 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152340 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152435    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.152497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152557    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152580    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.152765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152781    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152794    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwt6v\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152820    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152854    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.152902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config\") pod \"1961650c-0655-4f79-b8e5-5a288796228d\" (UID: \"1961650c-0655-4f79-b8e5-5a288796228d\") "
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.153174    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.153190    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.153305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.153202    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-metrics-client-ca\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.153884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.153680    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.155317 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.154179    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:20:33.155317 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.154738    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.155829 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.155801    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.156103 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.156073    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.156338 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.156311    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.156516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.156493    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.156678 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.156640    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:20:33.157129 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.156827    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config" (OuterVolumeSpecName: "config") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.157129 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.157094    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:20:33.157945 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.157831    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out" (OuterVolumeSpecName: "config-out") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:20:33.158106 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.158008    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.158233 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.158150    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.158584 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.158558    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.159100 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.159073    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.159469 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.159443    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v" (OuterVolumeSpecName: "kube-api-access-nwt6v") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "kube-api-access-nwt6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:20:33.168698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.168667    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config" (OuterVolumeSpecName: "web-config") pod "1961650c-0655-4f79-b8e5-5a288796228d" (UID: "1961650c-0655-4f79-b8e5-5a288796228d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216533    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566" exitCode=0
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216558    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f" exitCode=0
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216564    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d" exitCode=0
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216570    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031" exitCode=0
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216574    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995" exitCode=0
Jul 07 02:20:33.216571 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216581    2561 generic.go:356] "Generic (PLEG): container finished" podID="1961650c-0655-4f79-b8e5-5a288796228d" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6" exitCode=0
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216638    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216826    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"}
Jul 07 02:20:33.217241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.216947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1961650c-0655-4f79-b8e5-5a288796228d","Type":"ContainerDied","Data":"586834f0c8d404f7fba202f14bfd09e32d12ae915a5439df6990b29e7e23e62f"}
Jul 07 02:20:33.225725 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.225705    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.233145 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.233127    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.240809 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.240786    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.241716 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.241692    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:33.245785 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.245762    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:33.250711 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.250692    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.254273 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254252    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254277    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254293    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254306    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-kube-rbac-proxy\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254319    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwt6v\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-kube-api-access-nwt6v\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254331    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-metrics-client-certs\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254345    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254365 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254358    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254372    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1961650c-0655-4f79-b8e5-5a288796228d-tls-assets\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254387    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254401    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254414    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-config-out\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254428    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-db\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254441    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-secret-grpc-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254454    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254468    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1961650c-0655-4f79-b8e5-5a288796228d-web-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.254648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.254481    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1961650c-0655-4f79-b8e5-5a288796228d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:20:33.258813 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.258791    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.267391 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.267369    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.275367 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275339    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:33.275670 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275672    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275680    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="config-reloader"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275686    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="config-reloader"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275693    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="init-config-reloader"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275699    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="init-config-reloader"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275708    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="prometheus"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275713    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="prometheus"
Jul 07 02:20:33.275727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275727    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-web"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275732    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-web"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275741    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-thanos"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275746    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-thanos"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275754    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="thanos-sidecar"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275759    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="thanos-sidecar"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275805    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-web"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275811    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275818    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="kube-rbac-proxy-thanos"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275825    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="thanos-sidecar"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275831    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="config-reloader"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275837    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1961650c-0655-4f79-b8e5-5a288796228d" containerName="prometheus"
Jul 07 02:20:33.276130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.275919    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.276616 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.276266    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.276616 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276304    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.276616 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276332    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.276729 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.276646    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.276729 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276670    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.276729 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276686    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.276951 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.276936    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.276989 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276954    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.276989 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.276966    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.277182 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.277161    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.277245 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.277245 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277211    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.277463 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.277437    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.277530 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277462    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.277530 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277481    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.277724 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.277704    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.277777 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.277777 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277745    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.277953 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:20:33.277936    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.278019 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277959    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.278019 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.277979    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.278186 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278171    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.278233 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278186    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.278386 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278367    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.278461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278388    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.278732 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278712    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.278798 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278734    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.278968 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278951    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.279014 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.278968    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.279164 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279146    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.279211 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279166    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.279373 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279350    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.279454 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279372    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.279581 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.279652 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279582    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.279776 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279759    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.279830 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279777    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.279963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279946    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.280003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.279963    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.280165 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280145    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.280233 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280167    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.280407 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280390    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.280467 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280409    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.280467 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.280647 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280600    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.280689 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280647    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.280864 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280837    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.280942 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.280867    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.281129 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281096    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.281129 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281126    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.281552 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.281552 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281553    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.281843 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281825    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.281843 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.281842    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.282133 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282109    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.282198 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282136    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.282384 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282364    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.282384 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282383    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.282604 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282587    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.282696 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282603    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.282883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282859    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.282883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.282882    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.283139 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283121    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.283192 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283140    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.283419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283397    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.283419 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283417    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.283613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 02:20:33.283613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283421    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 02:20:33.283752 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283695    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.283752 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283716    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.283752 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283719    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 02:20:33.283906 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283803    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cd42j\""
Jul 07 02:20:33.283906 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283836    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 02:20:33.283906 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283840    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 02:20:33.284051 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283936    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 02:20:33.284051 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.283994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 02:20:33.284051 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284010    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 02:20:33.284175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akcdlv3424s7u\""
Jul 07 02:20:33.284175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 02:20:33.284175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284109    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 02:20:33.284323 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 02:20:33.284733 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284712    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.284733 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.284733    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.285047 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.285026    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.285047 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.285046    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.285204 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.285187    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 02:20:33.285699 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.285674    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.285762 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.285701    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.286031 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286000    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.286031 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286024    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.286309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286286    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.286382 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286310    2561 scope.go:117] "RemoveContainer" containerID="0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"
Jul 07 02:20:33.286647 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286595    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566"} err="failed to get container status \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": rpc error: code = NotFound desc = could not find container \"0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566\": container with ID starting with 0c6c517cf4a51f688e78f11511df93d51f5e86609206a2ad3a4d6b5c327c1566 not found: ID does not exist"
Jul 07 02:20:33.286647 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286646    2561 scope.go:117] "RemoveContainer" containerID="6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"
Jul 07 02:20:33.286825 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286807    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 02:20:33.286897 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286881    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f"} err="failed to get container status \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": rpc error: code = NotFound desc = could not find container \"6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f\": container with ID starting with 6fe8c76e09dcdf95c0e0e137896fdd355e238fb055de54fcd357847fb3d4014f not found: ID does not exist"
Jul 07 02:20:33.286955 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.286902    2561 scope.go:117] "RemoveContainer" containerID="cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"
Jul 07 02:20:33.287181 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287162    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d"} err="failed to get container status \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": rpc error: code = NotFound desc = could not find container \"cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d\": container with ID starting with cbfd78b49bf04d7b786f15c9d22300ab4f4918f4b4f7f9cc432e5291b7b1431d not found: ID does not exist"
Jul 07 02:20:33.287250 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287182    2561 scope.go:117] "RemoveContainer" containerID="83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"
Jul 07 02:20:33.287462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287433    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031"} err="failed to get container status \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": rpc error: code = NotFound desc = could not find container \"83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031\": container with ID starting with 83dc9f8e59721d2df78f117871aa32a0d4cbe5f3e9a8c4cd3c6060a3acf66031 not found: ID does not exist"
Jul 07 02:20:33.287542 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287463    2561 scope.go:117] "RemoveContainer" containerID="5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"
Jul 07 02:20:33.287738 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287702    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1961650c-0655-4f79-b8e5-5a288796228d" path="/var/lib/kubelet/pods/1961650c-0655-4f79-b8e5-5a288796228d/volumes"
Jul 07 02:20:33.287842 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287775    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995"} err="failed to get container status \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": rpc error: code = NotFound desc = could not find container \"5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995\": container with ID starting with 5d0e6be31b9671f9e2ff1cc5881f21510e1be94c1aa56c19cb94e231a4eee995 not found: ID does not exist"
Jul 07 02:20:33.287842 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.287791    2561 scope.go:117] "RemoveContainer" containerID="3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"
Jul 07 02:20:33.288224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.288189    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6"} err="failed to get container status \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": rpc error: code = NotFound desc = could not find container \"3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6\": container with ID starting with 3bfa3b69bdf900441cedc792f8b464f7089f853266159c8ebe19acca2e4771f6 not found: ID does not exist"
Jul 07 02:20:33.288224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.288212    2561 scope.go:117] "RemoveContainer" containerID="c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"
Jul 07 02:20:33.288497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.288479    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751"} err="failed to get container status \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": rpc error: code = NotFound desc = could not find container \"c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751\": container with ID starting with c8c122ff6bca819d130d5f0308851eeeb58c457d28c5b9348cf0bce74eafe751 not found: ID does not exist"
Jul 07 02:20:33.289677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.289659    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 02:20:33.299963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.299934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:33.355261 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355261 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355261 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355261 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355811 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355811 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.355912 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356016 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjs4\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-kube-api-access-xpjs4\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356016 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.355992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356087 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356125 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356165 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356165 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356235 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.356235 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.356225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.456834 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.456797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.456983 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.456842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.456972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457434    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjs4\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-kube-api-access-xpjs4\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.457903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.457521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.460015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.459120    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.460015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.459356    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.460015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.459934    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.460343 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.460316    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.460483 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.460456    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461030 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.460924    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461030 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.460926    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.461054    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461180 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.461092    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461344 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.461289    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa0679bb-9a22-43f2-97eb-e9bd0556624a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461726 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.461666    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.461894 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.461867    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.462461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.462375    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.462461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.462424    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-config\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.462766 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.462748    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.462886 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.462869    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.466297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.463682    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.466297 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.463699    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa0679bb-9a22-43f2-97eb-e9bd0556624a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.469280 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.469252    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjs4\" (UniqueName: \"kubernetes.io/projected/aa0679bb-9a22-43f2-97eb-e9bd0556624a-kube-api-access-xpjs4\") pod \"prometheus-k8s-0\" (UID: \"aa0679bb-9a22-43f2-97eb-e9bd0556624a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.592104 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.592064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:20:33.736747 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:33.736717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:20:33.738700 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:20:33.738672    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0679bb_9a22_43f2_97eb_e9bd0556624a.slice/crio-b2e76c8b6227f9e08150f85aab997434539ecf44e0eccfe1c7e280748d417262 WatchSource:0}: Error finding container b2e76c8b6227f9e08150f85aab997434539ecf44e0eccfe1c7e280748d417262: Status 404 returned error can't find the container with id b2e76c8b6227f9e08150f85aab997434539ecf44e0eccfe1c7e280748d417262
Jul 07 02:20:34.222503 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:34.222469    2561 generic.go:356] "Generic (PLEG): container finished" podID="aa0679bb-9a22-43f2-97eb-e9bd0556624a" containerID="922af780a2ab308d7543f7179537f51253c383237efc33a55e87cc73db912dab" exitCode=0
Jul 07 02:20:34.222958 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:34.222566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerDied","Data":"922af780a2ab308d7543f7179537f51253c383237efc33a55e87cc73db912dab"}
Jul 07 02:20:34.222958 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:34.222661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"b2e76c8b6227f9e08150f85aab997434539ecf44e0eccfe1c7e280748d417262"}
Jul 07 02:20:35.232445 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"06282b26873da446d1779003392cde9f4020e8cf0476551753e9555ee210dda4"}
Jul 07 02:20:35.232969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"c503a2d8e8ad64cfaf1f297d0ae90464f935c6a094f3c0d54225d42a96656ec1"}
Jul 07 02:20:35.232969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"0d1429583300674ae6e0433090a98cc73a122a564e4bac23683010836fb3a4a6"}
Jul 07 02:20:35.232969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"c4fbf4b2902b9e142a7c44163a921b4ce1489e888de70c68230bdf1cb633fc85"}
Jul 07 02:20:35.232969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"1392ee6836f0f44cee135bf4e00e0041a70d36936fb7c66ab7653712e92a13c4"}
Jul 07 02:20:35.232969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.232566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa0679bb-9a22-43f2-97eb-e9bd0556624a","Type":"ContainerStarted","Data":"6562367f24893341d69ba918925e6485695e7969cb36ff3c4230548f8f0326c8"}
Jul 07 02:20:35.264284 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:35.264230    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.245378291 podStartE2EDuration="2.264206841s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.82855ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:20:35.261348549 +0000 UTC m=+146.568666939" watchObservedRunningTime="2026-07-07 02:20:35.264206841 +0000 UTC m=+146.571525245"
Jul 07 02:20:38.592511 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:20:38.592462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:21:33.592349 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:21:33.592314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:21:33.608392 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:21:33.608367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:21:34.477244 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:21:34.477214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:09.119376 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:23:09.119343    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:23:09.126137 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:23:09.126109    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:23:09.126763 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:23:09.126747    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 02:24:01.117888 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.117811    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2ck2p"]
Jul 07 02:24:01.120005 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.119988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.122829 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.122807    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 02:24:01.122971 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.122807    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 02:24:01.123247 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.123230    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 02:24:01.124041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.124003    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mq9sn\""
Jul 07 02:24:01.140465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.140439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ck2p"]
Jul 07 02:24:01.258048 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.258009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbctd\" (UniqueName: \"kubernetes.io/projected/c17eef29-ca4b-4558-9de2-28f0b21d18fb-kube-api-access-rbctd\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.258229 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.258062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c17eef29-ca4b-4558-9de2-28f0b21d18fb-data\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.359254 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.359212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c17eef29-ca4b-4558-9de2-28f0b21d18fb-data\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.359471 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.359376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbctd\" (UniqueName: \"kubernetes.io/projected/c17eef29-ca4b-4558-9de2-28f0b21d18fb-kube-api-access-rbctd\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.359685 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.359663    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c17eef29-ca4b-4558-9de2-28f0b21d18fb-data\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.368448 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.368393    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbctd\" (UniqueName: \"kubernetes.io/projected/c17eef29-ca4b-4558-9de2-28f0b21d18fb-kube-api-access-rbctd\") pod \"seaweedfs-64568bcd49-2ck2p\" (UID: \"c17eef29-ca4b-4558-9de2-28f0b21d18fb\") " pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.429550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.429517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:01.559610 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.559584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ck2p"]
Jul 07 02:24:01.562259 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:24:01.562236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17eef29_ca4b_4558_9de2_28f0b21d18fb.slice/crio-2c430849337da5f589ba8d6bd447a7841a98ba1ba1a7b7b00bd1443c5f983df0 WatchSource:0}: Error finding container 2c430849337da5f589ba8d6bd447a7841a98ba1ba1a7b7b00bd1443c5f983df0: Status 404 returned error can't find the container with id 2c430849337da5f589ba8d6bd447a7841a98ba1ba1a7b7b00bd1443c5f983df0
Jul 07 02:24:01.563491 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:01.563475    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:24:02.030672 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:02.030615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ck2p" event={"ID":"c17eef29-ca4b-4558-9de2-28f0b21d18fb","Type":"ContainerStarted","Data":"2c430849337da5f589ba8d6bd447a7841a98ba1ba1a7b7b00bd1443c5f983df0"}
Jul 07 02:24:06.049774 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:06.049742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ck2p" event={"ID":"c17eef29-ca4b-4558-9de2-28f0b21d18fb","Type":"ContainerStarted","Data":"b7e30660850691d63c2be46494f7294fe01a94744c4be7dd030802ae50b5c2f7"}
Jul 07 02:24:06.050141 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:06.049835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:06.066784 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:06.066738    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2ck2p" podStartSLOduration=1.452487507 podStartE2EDuration="5.066722199s" totalImagesPullingTime="3.614234692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:06.065690537 +0000 UTC m=+357.373008925" watchObservedRunningTime="2026-07-07 02:24:06.066722199 +0000 UTC m=+357.374040589"
Jul 07 02:24:12.056503 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:12.056473    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2ck2p"
Jul 07 02:24:32.497992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.497952    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-745b8bd5f6-rk2wc"]
Jul 07 02:24:32.500857 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.500834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503324 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47sf\" (UniqueName: \"kubernetes.io/projected/8eb437d2-e558-4e41-8ffd-21a80878e9f2-kube-api-access-z47sf\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503424 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-service-ca\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503424 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-trusted-ca-bundle\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503424 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503530 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503530 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-oauth-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.503604 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.503588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-oauth-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.504164 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504145    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 02:24:32.504426 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504411    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 02:24:32.504509 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504425    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 02:24:32.504509 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504453    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 02:24:32.504948 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 02:24:32.505131 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.504969    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 02:24:32.505131 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.505038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mmk7l\""
Jul 07 02:24:32.505769 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.505748    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 02:24:32.509257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.509240    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 02:24:32.516758 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.516741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745b8bd5f6-rk2wc"]
Jul 07 02:24:32.604550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-trusted-ca-bundle\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-oauth-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-oauth-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47sf\" (UniqueName: \"kubernetes.io/projected/8eb437d2-e558-4e41-8ffd-21a80878e9f2-kube-api-access-z47sf\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.604841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.604768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-service-ca\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.605436 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.605409    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-oauth-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.605529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.605442    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-service-ca\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.605569 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.605535    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.605683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.605668    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb437d2-e558-4e41-8ffd-21a80878e9f2-trusted-ca-bundle\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.607106 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.607091    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-oauth-config\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.607208 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.607194    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb437d2-e558-4e41-8ffd-21a80878e9f2-console-serving-cert\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.613117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.613092    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47sf\" (UniqueName: \"kubernetes.io/projected/8eb437d2-e558-4e41-8ffd-21a80878e9f2-kube-api-access-z47sf\") pod \"console-745b8bd5f6-rk2wc\" (UID: \"8eb437d2-e558-4e41-8ffd-21a80878e9f2\") " pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.810699 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.810663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:32.944988 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:32.944961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745b8bd5f6-rk2wc"]
Jul 07 02:24:32.947466 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:24:32.947436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb437d2_e558_4e41_8ffd_21a80878e9f2.slice/crio-63034aa7d30c52a46b64c7a414a4bd9b51615ef7f2c67da1203e5814f417f0c2 WatchSource:0}: Error finding container 63034aa7d30c52a46b64c7a414a4bd9b51615ef7f2c67da1203e5814f417f0c2: Status 404 returned error can't find the container with id 63034aa7d30c52a46b64c7a414a4bd9b51615ef7f2c67da1203e5814f417f0c2
Jul 07 02:24:33.157525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:33.157429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745b8bd5f6-rk2wc" event={"ID":"8eb437d2-e558-4e41-8ffd-21a80878e9f2","Type":"ContainerStarted","Data":"58dd97553b7ef61c485949e4445f1eae64b2cdbd55a7bcb941f06a0d08a2a311"}
Jul 07 02:24:33.157525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:33.157509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745b8bd5f6-rk2wc" event={"ID":"8eb437d2-e558-4e41-8ffd-21a80878e9f2","Type":"ContainerStarted","Data":"63034aa7d30c52a46b64c7a414a4bd9b51615ef7f2c67da1203e5814f417f0c2"}
Jul 07 02:24:33.176030 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:33.175981    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-745b8bd5f6-rk2wc" podStartSLOduration=1.175965558 podStartE2EDuration="1.175965558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:33.174837418 +0000 UTC m=+384.482155832" watchObservedRunningTime="2026-07-07 02:24:33.175965558 +0000 UTC m=+384.483283949"
Jul 07 02:24:42.811122 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:42.811085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:42.811523 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:42.811359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:42.815812 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:42.815790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:24:43.201458 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:24:43.201384    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-745b8bd5f6-rk2wc"
Jul 07 02:25:11.146187 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.146152    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-4n7f4"]
Jul 07 02:25:11.148706 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.148684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.151301 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.151271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 02:25:11.151508 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.151490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-wfknc\""
Jul 07 02:25:11.158342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.158315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-4n7f4"]
Jul 07 02:25:11.161000 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.160980    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-np8c8"]
Jul 07 02:25:11.163381 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.163365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.166270 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.166250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 02:25:11.166411 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.166396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-fw5wm\""
Jul 07 02:25:11.172919 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.172897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-np8c8"]
Jul 07 02:25:11.235944 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.235905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.236121 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.236045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4bc\" (UniqueName: \"kubernetes.io/projected/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-kube-api-access-8h4bc\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.337505 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.337471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4bc\" (UniqueName: \"kubernetes.io/projected/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-kube-api-access-8h4bc\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.337714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.337514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.337714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.337554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/043105df-4a88-4c68-9f42-3f92c756fd69-cert\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.337714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.337571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ll6\" (UniqueName: \"kubernetes.io/projected/043105df-4a88-4c68-9f42-3f92c756fd69-kube-api-access-s4ll6\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.337855 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:25:11.337719    2561 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 02:25:11.337855 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:25:11.337770    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs podName:8fc1061f-bd4d-418b-a3d4-03457fb9a16d nodeName:}" failed. No retries permitted until 2026-07-07 02:25:11.837754343 +0000 UTC m=+423.145072711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs") pod "model-serving-api-fd65d7d6-4n7f4" (UID: "8fc1061f-bd4d-418b-a3d4-03457fb9a16d") : secret "model-serving-api-tls" not found
Jul 07 02:25:11.346534 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.346507    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4bc\" (UniqueName: \"kubernetes.io/projected/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-kube-api-access-8h4bc\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.438390 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.438308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/043105df-4a88-4c68-9f42-3f92c756fd69-cert\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.438390 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.438339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ll6\" (UniqueName: \"kubernetes.io/projected/043105df-4a88-4c68-9f42-3f92c756fd69-kube-api-access-s4ll6\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.440950 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.440928    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/043105df-4a88-4c68-9f42-3f92c756fd69-cert\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.446765 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.446745    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ll6\" (UniqueName: \"kubernetes.io/projected/043105df-4a88-4c68-9f42-3f92c756fd69-kube-api-access-s4ll6\") pod \"odh-model-controller-84b54b7f97-np8c8\" (UID: \"043105df-4a88-4c68-9f42-3f92c756fd69\") " pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.473810 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.473777    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:11.600651 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.600584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-np8c8"]
Jul 07 02:25:11.603148 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:25:11.603125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043105df_4a88_4c68_9f42_3f92c756fd69.slice/crio-57151194e10dcf492de2ed611aa293acb117202532f2a613c9486fc89c140fb4 WatchSource:0}: Error finding container 57151194e10dcf492de2ed611aa293acb117202532f2a613c9486fc89c140fb4: Status 404 returned error can't find the container with id 57151194e10dcf492de2ed611aa293acb117202532f2a613c9486fc89c140fb4
Jul 07 02:25:11.842138 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.842105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:11.844612 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:11.844587    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc1061f-bd4d-418b-a3d4-03457fb9a16d-tls-certs\") pod \"model-serving-api-fd65d7d6-4n7f4\" (UID: \"8fc1061f-bd4d-418b-a3d4-03457fb9a16d\") " pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:12.059748 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:12.059714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:12.193024 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:12.192987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-4n7f4"]
Jul 07 02:25:12.197068 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:25:12.197036    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc1061f_bd4d_418b_a3d4_03457fb9a16d.slice/crio-9b813f8bc52ff295a49e023a07fc78b13c660c71753d111107c055725b1d5b7f WatchSource:0}: Error finding container 9b813f8bc52ff295a49e023a07fc78b13c660c71753d111107c055725b1d5b7f: Status 404 returned error can't find the container with id 9b813f8bc52ff295a49e023a07fc78b13c660c71753d111107c055725b1d5b7f
Jul 07 02:25:12.315654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:12.315598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-np8c8" event={"ID":"043105df-4a88-4c68-9f42-3f92c756fd69","Type":"ContainerStarted","Data":"57151194e10dcf492de2ed611aa293acb117202532f2a613c9486fc89c140fb4"}
Jul 07 02:25:12.316832 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:12.316808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-4n7f4" event={"ID":"8fc1061f-bd4d-418b-a3d4-03457fb9a16d","Type":"ContainerStarted","Data":"9b813f8bc52ff295a49e023a07fc78b13c660c71753d111107c055725b1d5b7f"}
Jul 07 02:25:16.338895 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.338854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-np8c8" event={"ID":"043105df-4a88-4c68-9f42-3f92c756fd69","Type":"ContainerStarted","Data":"dd4817a0b50138196ee36dc6b3e7b932be0885bb9167e2402b908aa436a893bb"}
Jul 07 02:25:16.339334 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.339057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:16.340387 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.340365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-4n7f4" event={"ID":"8fc1061f-bd4d-418b-a3d4-03457fb9a16d","Type":"ContainerStarted","Data":"93cb1e10333b52367a753ed92603d57bdf55814c0245729cb9dbf06abce10165"}
Jul 07 02:25:16.340500 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.340489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:16.355923 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.355882    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-np8c8" podStartSLOduration=1.314987655 podStartE2EDuration="5.355868502s" totalImagesPullingTime="4.040880847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:16.354142002 +0000 UTC m=+427.661460397" watchObservedRunningTime="2026-07-07 02:25:16.355868502 +0000 UTC m=+427.663186892"
Jul 07 02:25:16.371656 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:16.371589    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-4n7f4" podStartSLOduration=1.887461531 podStartE2EDuration="5.371578737s" totalImagesPullingTime="3.484117206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:16.370800662 +0000 UTC m=+427.678119052" watchObservedRunningTime="2026-07-07 02:25:16.371578737 +0000 UTC m=+427.678897127"
Jul 07 02:25:27.347892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:27.347864    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-np8c8"
Jul 07 02:25:27.349585 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:27.349566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-4n7f4"
Jul 07 02:25:28.180391 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.180358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cfgcw"]
Jul 07 02:25:28.182986 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.182970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.191911 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.191887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cfgcw"]
Jul 07 02:25:28.286966 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.286935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.286966 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.286972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbgf\" (UniqueName: \"kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.388416 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.388378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.388909 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.388470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbgf\" (UniqueName: \"kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.389115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.388815    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.397493 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.397469    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbgf\" (UniqueName: \"kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf\") pod \"s3-init-cfgcw\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") " pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.510276 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.510196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cfgcw"
Jul 07 02:25:28.645359 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:28.645327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cfgcw"]
Jul 07 02:25:28.647728 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:25:28.647691    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69cbb55_84ad_4e5a_bd7a_f8643981f5e6.slice/crio-c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e WatchSource:0}: Error finding container c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e: Status 404 returned error can't find the container with id c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e
Jul 07 02:25:29.402561 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:29.402470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerStarted","Data":"c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e"}
Jul 07 02:25:32.418558 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:32.418526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerStarted","Data":"2404290125b5ad85e1d1d277836a87f0362023673f20d29c8df9e1282dd77602"}
Jul 07 02:25:34.430545 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:34.430455    2561 generic.go:356] "Generic (PLEG): container finished" podID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerID="2404290125b5ad85e1d1d277836a87f0362023673f20d29c8df9e1282dd77602" exitCode=0
Jul 07 02:25:34.430929 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:34.430533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerDied","Data":"2404290125b5ad85e1d1d277836a87f0362023673f20d29c8df9e1282dd77602"}
Jul 07 02:25:39.456492 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:39.456459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerStarted","Data":"9a44322ed21dd6fa41596ea6990124d186e3e8307c34361552550cc89ea8a7af"}
Jul 07 02:25:39.475709 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:25:39.475660    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cfgcw" podStartSLOduration=1.513265962 podStartE2EDuration="11.475616505s" totalImagesPullingTime="7.815974256s" totalInitContainerRuntime="2.146376287s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:25:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:39.474494235 +0000 UTC m=+450.781812626" watchObservedRunningTime="2026-07-07 02:25:39.475616505 +0000 UTC m=+450.782934894"
Jul 07 02:26:44.734737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:44.734697    2561 generic.go:356] "Generic (PLEG): container finished" podID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerID="9a44322ed21dd6fa41596ea6990124d186e3e8307c34361552550cc89ea8a7af" exitCode=0
Jul 07 02:26:44.734737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:44.734742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerDied","Data":"9a44322ed21dd6fa41596ea6990124d186e3e8307c34361552550cc89ea8a7af"}
Jul 07 02:26:45.872439 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:45.872411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cfgcw"
Jul 07 02:26:45.902442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:45.902401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache\") pod \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") "
Jul 07 02:26:45.902603 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:45.902464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxbgf\" (UniqueName: \"kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf\") pod \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\" (UID: \"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6\") "
Jul 07 02:26:45.904859 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:45.904821    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf" (OuterVolumeSpecName: "kube-api-access-pxbgf") pod "e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" (UID: "e69cbb55-84ad-4e5a-bd7a-f8643981f5e6"). InnerVolumeSpecName "kube-api-access-pxbgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:26:45.921610 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:45.921580    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" (UID: "e69cbb55-84ad-4e5a-bd7a-f8643981f5e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:26:46.004144 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:46.004058    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-model-cache\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:26:46.004144 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:46.004088    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxbgf\" (UniqueName: \"kubernetes.io/projected/e69cbb55-84ad-4e5a-bd7a-f8643981f5e6-kube-api-access-pxbgf\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:26:46.744578 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:46.744549    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cfgcw"
Jul 07 02:26:46.744807 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:46.744582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cfgcw" event={"ID":"e69cbb55-84ad-4e5a-bd7a-f8643981f5e6","Type":"ContainerDied","Data":"c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e"}
Jul 07 02:26:46.744807 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:46.744639    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e357e3a15524386f4b593716851368361c71290597a74b45c99882ec5daa6e"
Jul 07 02:26:58.177231 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177195    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:26:58.177607 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerName="s3-init"
Jul 07 02:26:58.177607 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177536    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerName="s3-init"
Jul 07 02:26:58.177607 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177562    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerName="download-hf-model"
Jul 07 02:26:58.177607 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177568    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerName="download-hf-model"
Jul 07 02:26:58.177768 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.177635    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69cbb55-84ad-4e5a-bd7a-f8643981f5e6" containerName="s3-init"
Jul 07 02:26:58.183124 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.183104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.185841 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.185815    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2346e-predictor-serving-cert\""
Jul 07 02:26:58.185965 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.185912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:26:58.186055 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.185974    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2346e-kube-rbac-proxy-sar-config\""
Jul 07 02:26:58.187165 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.187149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:26:58.187293 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.187261    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gqpvr\""
Jul 07 02:26:58.189719 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.189698    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:26:58.312930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.312896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.313100 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.312955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2nm\" (UniqueName: \"kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.313100 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.312981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.414380 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.414345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2nm\" (UniqueName: \"kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.414550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.414386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.414550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.414452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.415263 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.415243    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.417151 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.417129    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.422794 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.422761    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2nm\" (UniqueName: \"kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm\") pod \"success-200-isvc-2346e-predictor-f9b49f577-z2m5r\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") " pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.465061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.464970    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:26:58.467887 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.467868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.470660 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.470633    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 07 02:26:58.470799 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.470726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 07 02:26:58.470799 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.470781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:26:58.470917 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.470782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:26:58.478407 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.478383    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:26:58.494030 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.494002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:26:58.515639 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.515598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.515801 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.515663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.515801 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.515693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.515801 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.515710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.515930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.515849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j55\" (UniqueName: \"kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.616322 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.616292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.616518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.616340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.616518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.616370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.616518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.616394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.616518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.616479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j55\" (UniqueName: \"kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.617476 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:58.617333    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 02:26:58.617476 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:58.617403    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls podName:e17e9087-204f-4750-9bf3-762232f66d66 nodeName:}" failed. No retries permitted until 2026-07-07 02:26:59.117381721 +0000 UTC m=+530.424700093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls") pod "isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" (UID: "e17e9087-204f-4750-9bf3-762232f66d66") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 02:26:58.617476 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.617425    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.617683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.617515    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.617683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.617532    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.629792 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.629765    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j55\" (UniqueName: \"kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:58.654196 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.654170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:26:58.656786 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:26:58.656755    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5392cc5b_e830_40d9_b4d1_e673cd917124.slice/crio-2293cf5b26f5fbc529bf6054e8a50615cbc7bf8229c466a76d5fe26c76471978 WatchSource:0}: Error finding container 2293cf5b26f5fbc529bf6054e8a50615cbc7bf8229c466a76d5fe26c76471978: Status 404 returned error can't find the container with id 2293cf5b26f5fbc529bf6054e8a50615cbc7bf8229c466a76d5fe26c76471978
Jul 07 02:26:58.679613 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.679586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:26:58.682992 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.682973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.685833 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.685808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 07 02:26:58.685961 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.685811    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 07 02:26:58.695690 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.695666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:26:58.717147 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.717053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.717147 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.717109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.717331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.717178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnlb\" (UniqueName: \"kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.717331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.717266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.717331 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.717298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.776238 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.776194    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:26:58.779071 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.779054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.781794 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.781770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 07 02:26:58.781990 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.781809    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 07 02:26:58.788997 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.788970    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:26:58.797115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.797089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerStarted","Data":"2293cf5b26f5fbc529bf6054e8a50615cbc7bf8229c466a76d5fe26c76471978"}
Jul 07 02:26:58.818455 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.818455 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnlb\" (UniqueName: \"kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.818728 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.819041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.819041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.818839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.819140 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.819075    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.819479 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.819458    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.819514 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.819458    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.821055 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.821038    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.826934 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.826915    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnlb\" (UniqueName: \"kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb\") pod \"isvc-xgboost-graph-predictor-555fc69d56-vbstl\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:58.920137 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920137 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920370 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920370 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:58.920231    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 02:26:58.920370 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:58.920303    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls podName:2635b90c-7986-46a8-a45a-1a209eb475fb nodeName:}" failed. No retries permitted until 2026-07-07 02:26:59.420283372 +0000 UTC m=+530.727601740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls") pod "isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 02:26:58.920370 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920553 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920553 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920529    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.920971 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.920931    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.921375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.921358    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.928903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.928872    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:58.994162 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:58.994070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:26:59.121933 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.121896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:26:59.122127 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:59.122031    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 02:26:59.122127 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:26:59.122104    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls podName:e17e9087-204f-4750-9bf3-762232f66d66 nodeName:}" failed. No retries permitted until 2026-07-07 02:27:00.122087248 +0000 UTC m=+531.429405618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls") pod "isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" (UID: "e17e9087-204f-4750-9bf3-762232f66d66") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 02:26:59.133998 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.133945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:26:59.136762 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:26:59.136731    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636ca02f_d3be_454e_81ab_557b10492a29.slice/crio-bdba8ec704bf8473641c973140864860d5a847718fdf0619ada765eec7c110af WatchSource:0}: Error finding container bdba8ec704bf8473641c973140864860d5a847718fdf0619ada765eec7c110af: Status 404 returned error can't find the container with id bdba8ec704bf8473641c973140864860d5a847718fdf0619ada765eec7c110af
Jul 07 02:26:59.426054 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.425507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:59.445131 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.445048    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:59.690391 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.690298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:26:59.808305 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.808261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerStarted","Data":"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"}
Jul 07 02:26:59.808485 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.808383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerStarted","Data":"bdba8ec704bf8473641c973140864860d5a847718fdf0619ada765eec7c110af"}
Jul 07 02:26:59.973463 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:26:59.973314    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:26:59.976894 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:26:59.976858    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2635b90c_7986_46a8_a45a_1a209eb475fb.slice/crio-c616b3ff257c59868c990b81db5e14b30d22b369070704a6d11f826eedeb3f5d WatchSource:0}: Error finding container c616b3ff257c59868c990b81db5e14b30d22b369070704a6d11f826eedeb3f5d: Status 404 returned error can't find the container with id c616b3ff257c59868c990b81db5e14b30d22b369070704a6d11f826eedeb3f5d
Jul 07 02:27:00.136599 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.136464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:00.139710 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.139646    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:00.280859 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.280760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:00.510683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.510652    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:27:00.517576 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:27:00.517540    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17e9087_204f_4750_9bf3_762232f66d66.slice/crio-5332929aff0fbd3c5a8cba7553d93d83aec731e0ea3c66e1f36bf9d9590325f9 WatchSource:0}: Error finding container 5332929aff0fbd3c5a8cba7553d93d83aec731e0ea3c66e1f36bf9d9590325f9: Status 404 returned error can't find the container with id 5332929aff0fbd3c5a8cba7553d93d83aec731e0ea3c66e1f36bf9d9590325f9
Jul 07 02:27:00.822595 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.822469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerStarted","Data":"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"}
Jul 07 02:27:00.822595 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.822550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerStarted","Data":"5332929aff0fbd3c5a8cba7553d93d83aec731e0ea3c66e1f36bf9d9590325f9"}
Jul 07 02:27:00.827658 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.827036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerStarted","Data":"4cbb7bc0ca6457a7c973afaad9f7da4e3ab76f4ce4cb42b3110a9c7354bf82b5"}
Jul 07 02:27:00.827658 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:00.827103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerStarted","Data":"c616b3ff257c59868c990b81db5e14b30d22b369070704a6d11f826eedeb3f5d"}
Jul 07 02:27:04.855028 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.854986    2561 generic.go:356] "Generic (PLEG): container finished" podID="636ca02f-d3be-454e-81ab-557b10492a29" containerID="18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7" exitCode=0
Jul 07 02:27:04.855507 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.855072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerDied","Data":"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"}
Jul 07 02:27:04.856984 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.856954    2561 generic.go:356] "Generic (PLEG): container finished" podID="e17e9087-204f-4750-9bf3-762232f66d66" containerID="97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3" exitCode=0
Jul 07 02:27:04.857082 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.857035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerDied","Data":"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"}
Jul 07 02:27:04.859287 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.859261    2561 generic.go:356] "Generic (PLEG): container finished" podID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerID="4cbb7bc0ca6457a7c973afaad9f7da4e3ab76f4ce4cb42b3110a9c7354bf82b5" exitCode=0
Jul 07 02:27:04.859367 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:04.859293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerDied","Data":"4cbb7bc0ca6457a7c973afaad9f7da4e3ab76f4ce4cb42b3110a9c7354bf82b5"}
Jul 07 02:27:15.931699 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:15.931584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerStarted","Data":"8da9a19ac28e8edb345ff21bff7f10fbaded9e6f8c3e63f9a2662a88e17cbfd7"}
Jul 07 02:27:15.933840 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:15.933812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerStarted","Data":"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"}
Jul 07 02:27:16.954946 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:16.954846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerStarted","Data":"230da3a852eeaa45c1ddb1c106281941b9a21f641766221c9da12a0acc154ebd"}
Jul 07 02:27:20.983335 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.983293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerStarted","Data":"36e72e3b4fcf5f1cedf7d3f2145f0fd5d5caaf3447ab1daac54efa36b8e88419"}
Jul 07 02:27:20.983874 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.983372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:27:20.983874 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.983485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:27:20.985313 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.985257    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:20.986168 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.986140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerStarted","Data":"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"}
Jul 07 02:27:20.986300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.986261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:20.988856 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.988829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerStarted","Data":"dc50ec0c61a6f795e70b3449f2ff35fb6f93ce2f04df22393b71cfeb6fe3a830"}
Jul 07 02:27:20.989046 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.989026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:27:20.989117 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.989068    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:27:20.990335 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:20.990308    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:21.006223 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.006170    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podStartSLOduration=1.801746644 podStartE2EDuration="23.006152954s" totalImagesPullingTime="21.20440631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:21.003599572 +0000 UTC m=+552.310917966" watchObservedRunningTime="2026-07-07 02:27:21.006152954 +0000 UTC m=+552.313471344"
Jul 07 02:27:21.034688 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.034044    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podStartSLOduration=4.046719347 podStartE2EDuration="23.034022254s" totalImagesPullingTime="15.005242279s" totalInitContainerRuntime="3.982060628s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:21.031422464 +0000 UTC m=+552.338740856" watchObservedRunningTime="2026-07-07 02:27:21.034022254 +0000 UTC m=+552.341340646"
Jul 07 02:27:21.056054 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.055998    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podStartSLOduration=3.950323374 podStartE2EDuration="23.055980369s" totalImagesPullingTime="15.015213807s" totalInitContainerRuntime="4.090443188s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:21.055596763 +0000 UTC m=+552.362915153" watchObservedRunningTime="2026-07-07 02:27:21.055980369 +0000 UTC m=+552.363298758"
Jul 07 02:27:21.995868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.995827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:21.995868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.995853    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:21.996462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.995933    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:21.996957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:21.996932    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:27:23.001115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:23.001076    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:27:27.001185 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:27.001138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:27:27.001674 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:27.001278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:27:27.001898 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:27.001840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:27.002145 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:27.002065    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:28.007300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:28.007272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:27:28.008062 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:28.008034    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:27:37.002185 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:37.002140    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:37.002185 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:37.002177    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:38.008097 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.008056    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:27:38.085121 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.085077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerStarted","Data":"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"}
Jul 07 02:27:38.085311 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.085193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerStarted","Data":"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"}
Jul 07 02:27:38.085569 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.085546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:27:38.085697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.085611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:27:38.087265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.087234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:27:38.119376 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:38.119319    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podStartSLOduration=3.68469822 podStartE2EDuration="40.11930231s" totalImagesPullingTime="32.215707258s" totalInitContainerRuntime="4.218896832s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:38.117015467 +0000 UTC m=+569.424333879" watchObservedRunningTime="2026-07-07 02:27:38.11930231 +0000 UTC m=+569.426620699"
Jul 07 02:27:39.090173 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:39.090132    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:27:44.094918 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:44.094884    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:27:44.095499 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:44.095473    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:27:47.001884 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:47.001839    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:47.002369 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:47.001840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:48.008706 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:48.008664    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:27:54.095906 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:54.095863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:27:57.002299 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:57.002252    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:27:57.002798 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:57.002252    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:27:58.008918 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:27:58.008871    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:28:04.096244 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:04.096205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:28:07.001661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:07.001603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 02:28:07.002140 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:07.001765    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:28:08.009149 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:08.009096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:28:09.157881 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:09.157854    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:28:09.160669 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:09.160647    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:28:14.096159 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:14.096114    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:28:17.001727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:17.001690    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:28:17.002405 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:17.002388    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:28:18.008747 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:18.008707    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:28:24.095709 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:24.095665    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:28:27.002608 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:27.002558    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:28:28.008876 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:28.008834    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:28:31.278938 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:31.278818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:28:34.095755 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:34.095716    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 02:28:38.008104 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.008062    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:28:38.689230 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.689189    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:28:38.689518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.689466    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container" containerID="cri-o://8da9a19ac28e8edb345ff21bff7f10fbaded9e6f8c3e63f9a2662a88e17cbfd7" gracePeriod=30
Jul 07 02:28:38.689518 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.689511    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kube-rbac-proxy" containerID="cri-o://36e72e3b4fcf5f1cedf7d3f2145f0fd5d5caaf3447ab1daac54efa36b8e88419" gracePeriod=30
Jul 07 02:28:38.758909 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.758872    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:28:38.761747 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.761729    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:38.765675 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.765652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3ea7f-predictor-serving-cert\""
Jul 07 02:28:38.765952 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.765938    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3ea7f-kube-rbac-proxy-sar-config\""
Jul 07 02:28:38.779950 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.779920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:28:38.920145 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.920102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:38.920333 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.920155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:38.920333 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:38.920242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.021094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.020983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.021094 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.021062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.021654 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.021115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.021704 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.021690    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.023863 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.023830    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.044352 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.044320    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r\") pod \"success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") " pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.072259 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.072220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.233106 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.233081    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:28:39.235540 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:28:39.235503    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd23a8eb_edd5_4d82_9523_36e7aff2e825.slice/crio-b8c6602ceacc4df1bb12baa07e98802b2d09a85620e7f0b0382f1803e3606b83 WatchSource:0}: Error finding container b8c6602ceacc4df1bb12baa07e98802b2d09a85620e7f0b0382f1803e3606b83: Status 404 returned error can't find the container with id b8c6602ceacc4df1bb12baa07e98802b2d09a85620e7f0b0382f1803e3606b83
Jul 07 02:28:39.398317 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.398285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerStarted","Data":"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"}
Jul 07 02:28:39.398517 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.398350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerStarted","Data":"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"}
Jul 07 02:28:39.398517 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.398379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerStarted","Data":"b8c6602ceacc4df1bb12baa07e98802b2d09a85620e7f0b0382f1803e3606b83"}
Jul 07 02:28:39.398517 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.398410    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.398517 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.398436    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:39.400225 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.400196    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:28:39.400423 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.400398    2561 generic.go:356] "Generic (PLEG): container finished" podID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerID="36e72e3b4fcf5f1cedf7d3f2145f0fd5d5caaf3447ab1daac54efa36b8e88419" exitCode=2
Jul 07 02:28:39.400474 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.400433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerDied","Data":"36e72e3b4fcf5f1cedf7d3f2145f0fd5d5caaf3447ab1daac54efa36b8e88419"}
Jul 07 02:28:39.423877 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:39.423830    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podStartSLOduration=1.423814327 podStartE2EDuration="1.423814327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:28:39.422445606 +0000 UTC m=+630.729763996" watchObservedRunningTime="2026-07-07 02:28:39.423814327 +0000 UTC m=+630.731132716"
Jul 07 02:28:40.280040 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:40.280009    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:28:40.406570 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:40.406533    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:28:41.284883 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:41.284858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:28:41.996939 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:41.996899    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 07 02:28:42.424293 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.424246    2561 generic.go:356] "Generic (PLEG): container finished" podID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerID="8da9a19ac28e8edb345ff21bff7f10fbaded9e6f8c3e63f9a2662a88e17cbfd7" exitCode=0
Jul 07 02:28:42.424745 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.424305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerDied","Data":"8da9a19ac28e8edb345ff21bff7f10fbaded9e6f8c3e63f9a2662a88e17cbfd7"}
Jul 07 02:28:42.446066 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.446040    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:28:42.450965 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.450936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm2nm\" (UniqueName: \"kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm\") pod \"5392cc5b-e830-40d9-b4d1-e673cd917124\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") "
Jul 07 02:28:42.451135 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.451044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config\") pod \"5392cc5b-e830-40d9-b4d1-e673cd917124\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") "
Jul 07 02:28:42.451135 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.451070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls\") pod \"5392cc5b-e830-40d9-b4d1-e673cd917124\" (UID: \"5392cc5b-e830-40d9-b4d1-e673cd917124\") "
Jul 07 02:28:42.451387 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.451358    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5392cc5b-e830-40d9-b4d1-e673cd917124" (UID: "5392cc5b-e830-40d9-b4d1-e673cd917124"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:28:42.453435 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.453404    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5392cc5b-e830-40d9-b4d1-e673cd917124" (UID: "5392cc5b-e830-40d9-b4d1-e673cd917124"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:28:42.453435 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.453423    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm" (OuterVolumeSpecName: "kube-api-access-cm2nm") pod "5392cc5b-e830-40d9-b4d1-e673cd917124" (UID: "5392cc5b-e830-40d9-b4d1-e673cd917124"). InnerVolumeSpecName "kube-api-access-cm2nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:28:42.552342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.552247    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:28:42.552342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.552283    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392cc5b-e830-40d9-b4d1-e673cd917124-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:28:42.552342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:42.552295    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2nm\" (UniqueName: \"kubernetes.io/projected/5392cc5b-e830-40d9-b4d1-e673cd917124-kube-api-access-cm2nm\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:28:43.431139 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.431106    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"
Jul 07 02:28:43.431139 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.431129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r" event={"ID":"5392cc5b-e830-40d9-b4d1-e673cd917124","Type":"ContainerDied","Data":"2293cf5b26f5fbc529bf6054e8a50615cbc7bf8229c466a76d5fe26c76471978"}
Jul 07 02:28:43.431666 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.431201    2561 scope.go:117] "RemoveContainer" containerID="36e72e3b4fcf5f1cedf7d3f2145f0fd5d5caaf3447ab1daac54efa36b8e88419"
Jul 07 02:28:43.440193 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.440165    2561 scope.go:117] "RemoveContainer" containerID="8da9a19ac28e8edb345ff21bff7f10fbaded9e6f8c3e63f9a2662a88e17cbfd7"
Jul 07 02:28:43.452679 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.452646    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:28:43.454680 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:43.454657    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2346e-predictor-f9b49f577-z2m5r"]
Jul 07 02:28:44.095786 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:44.095749    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:28:45.284150 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:45.284111    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" path="/var/lib/kubelet/pods/5392cc5b-e830-40d9-b4d1-e673cd917124/volumes"
Jul 07 02:28:45.411072 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:45.411044    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:28:45.411600 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:45.411569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:28:55.412484 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:28:55.412435    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:29:05.412153 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:05.412111    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:29:15.411905 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:15.411863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:29:18.494242 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.494208    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:29:18.494747 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.494445    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container" containerID="cri-o://f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6" gracePeriod=30
Jul 07 02:29:18.494747 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.494492    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kube-rbac-proxy" containerID="cri-o://44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3" gracePeriod=30
Jul 07 02:29:18.553995 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.553963    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:29:18.554282 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.554236    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" containerID="cri-o://230da3a852eeaa45c1ddb1c106281941b9a21f641766221c9da12a0acc154ebd" gracePeriod=30
Jul 07 02:29:18.554378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.554268    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kube-rbac-proxy" containerID="cri-o://dc50ec0c61a6f795e70b3449f2ff35fb6f93ce2f04df22393b71cfeb6fe3a830" gracePeriod=30
Jul 07 02:29:18.593483 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.593435    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:29:18.594115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594086    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container"
Jul 07 02:29:18.594115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594107    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container"
Jul 07 02:29:18.594554 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594385    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kube-rbac-proxy"
Jul 07 02:29:18.594554 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594400    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kube-rbac-proxy"
Jul 07 02:29:18.594554 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594509    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kserve-container"
Jul 07 02:29:18.594554 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.594524    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5392cc5b-e830-40d9-b4d1-e673cd917124" containerName="kube-rbac-proxy"
Jul 07 02:29:18.599081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.599034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.602782 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.602760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-821ea-predictor-serving-cert\""
Jul 07 02:29:18.603179 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.603162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-821ea-kube-rbac-proxy-sar-config\""
Jul 07 02:29:18.603336 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.603191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzccx\" (UniqueName: \"kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.603449 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.603430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.603520 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.603507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.606137 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.606112    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:29:18.606453 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.606427    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" containerID="cri-o://8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f" gracePeriod=30
Jul 07 02:29:18.606756 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.606488    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kube-rbac-proxy" containerID="cri-o://c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4" gracePeriod=30
Jul 07 02:29:18.611523 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.611310    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:29:18.704065 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.704030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.704265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.704108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzccx\" (UniqueName: \"kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.704265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.704174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.704265 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:18.704184    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-821ea-predictor-serving-cert: secret "success-200-isvc-821ea-predictor-serving-cert" not found
Jul 07 02:29:18.704265 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:18.704249    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls podName:9c913aa5-b07f-433f-8f22-f15f154b5c35 nodeName:}" failed. No retries permitted until 2026-07-07 02:29:19.204231511 +0000 UTC m=+670.511549880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls") pod "success-200-isvc-821ea-predictor-6597d66d67-9vj2t" (UID: "9c913aa5-b07f-433f-8f22-f15f154b5c35") : secret "success-200-isvc-821ea-predictor-serving-cert" not found
Jul 07 02:29:18.704867 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.704843    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:18.721272 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:18.721245    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzccx\" (UniqueName: \"kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:19.090791 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.090750    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 02:29:19.207519 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.207476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:19.210092 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.210058    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") pod \"success-200-isvc-821ea-predictor-6597d66d67-9vj2t\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") " pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:19.217949 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.217923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:19.366995 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.366912    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:29:19.370691 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:29:19.370665    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c913aa5_b07f_433f_8f22_f15f154b5c35.slice/crio-280900409d091ace93bae461aae8c606a907587c845571dc09963eee270fb8c6 WatchSource:0}: Error finding container 280900409d091ace93bae461aae8c606a907587c845571dc09963eee270fb8c6: Status 404 returned error can't find the container with id 280900409d091ace93bae461aae8c606a907587c845571dc09963eee270fb8c6
Jul 07 02:29:19.613521 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.613491    2561 generic.go:356] "Generic (PLEG): container finished" podID="636ca02f-d3be-454e-81ab-557b10492a29" containerID="44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3" exitCode=2
Jul 07 02:29:19.613953 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.613573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerDied","Data":"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"}
Jul 07 02:29:19.615645 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.615606    2561 generic.go:356] "Generic (PLEG): container finished" podID="e17e9087-204f-4750-9bf3-762232f66d66" containerID="c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4" exitCode=2
Jul 07 02:29:19.615739 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.615660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerDied","Data":"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"}
Jul 07 02:29:19.617704 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.617647    2561 generic.go:356] "Generic (PLEG): container finished" podID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerID="dc50ec0c61a6f795e70b3449f2ff35fb6f93ce2f04df22393b71cfeb6fe3a830" exitCode=2
Jul 07 02:29:19.617796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.617715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerDied","Data":"dc50ec0c61a6f795e70b3449f2ff35fb6f93ce2f04df22393b71cfeb6fe3a830"}
Jul 07 02:29:19.619432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.619404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerStarted","Data":"7fe17ea39038924f77ba89ff7f5f1ab0415179ce71af96dc561ffcd29afe7c1f"}
Jul 07 02:29:19.619544 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.619454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerStarted","Data":"4000c9e21c652e36dda27d8962a1de416118146c5e373b460014b5d53936d7a5"}
Jul 07 02:29:19.619544 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.619486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerStarted","Data":"280900409d091ace93bae461aae8c606a907587c845571dc09963eee270fb8c6"}
Jul 07 02:29:19.619544 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.619523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:19.641110 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:19.638477    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podStartSLOduration=1.638455279 podStartE2EDuration="1.638455279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:19.63701688 +0000 UTC m=+670.944335286" watchObservedRunningTime="2026-07-07 02:29:19.638455279 +0000 UTC m=+670.945773670"
Jul 07 02:29:20.279237 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:20.279197    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 02:29:20.623926 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:20.623895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:20.625300 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:20.625272    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:29:21.279150 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:21.279099    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 02:29:21.629039 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:21.628996    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:29:21.996527 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:21.996424    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 02:29:23.001318 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.001282    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 02:29:23.351235 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.351204    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:29:23.446819 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.446787    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xnlb\" (UniqueName: \"kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb\") pod \"636ca02f-d3be-454e-81ab-557b10492a29\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") "
Jul 07 02:29:23.447022 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.446836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config\") pod \"636ca02f-d3be-454e-81ab-557b10492a29\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") "
Jul 07 02:29:23.447022 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.446890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location\") pod \"636ca02f-d3be-454e-81ab-557b10492a29\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") "
Jul 07 02:29:23.447022 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.446930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls\") pod \"636ca02f-d3be-454e-81ab-557b10492a29\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") "
Jul 07 02:29:23.447022 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.446956    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert\") pod \"636ca02f-d3be-454e-81ab-557b10492a29\" (UID: \"636ca02f-d3be-454e-81ab-557b10492a29\") "
Jul 07 02:29:23.447253 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.447176    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "636ca02f-d3be-454e-81ab-557b10492a29" (UID: "636ca02f-d3be-454e-81ab-557b10492a29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:29:23.447253 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.447208    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "636ca02f-d3be-454e-81ab-557b10492a29" (UID: "636ca02f-d3be-454e-81ab-557b10492a29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:23.447378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.447357    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "636ca02f-d3be-454e-81ab-557b10492a29" (UID: "636ca02f-d3be-454e-81ab-557b10492a29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:23.449130 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.449108    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "636ca02f-d3be-454e-81ab-557b10492a29" (UID: "636ca02f-d3be-454e-81ab-557b10492a29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:29:23.449249 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.449171    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb" (OuterVolumeSpecName: "kube-api-access-7xnlb") pod "636ca02f-d3be-454e-81ab-557b10492a29" (UID: "636ca02f-d3be-454e-81ab-557b10492a29"). InnerVolumeSpecName "kube-api-access-7xnlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:29:23.547775 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.547688    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnlb\" (UniqueName: \"kubernetes.io/projected/636ca02f-d3be-454e-81ab-557b10492a29-kube-api-access-7xnlb\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:23.547775 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.547715    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:23.547775 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.547728    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/636ca02f-d3be-454e-81ab-557b10492a29-kserve-provision-location\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:23.547775 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.547737    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/636ca02f-d3be-454e-81ab-557b10492a29-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:23.547775 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.547747    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/636ca02f-d3be-454e-81ab-557b10492a29-cabundle-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:23.641816 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.641775    2561 generic.go:356] "Generic (PLEG): container finished" podID="636ca02f-d3be-454e-81ab-557b10492a29" containerID="f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6" exitCode=0
Jul 07 02:29:23.642023 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.641933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerDied","Data":"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"}
Jul 07 02:29:23.642023 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.641996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl" event={"ID":"636ca02f-d3be-454e-81ab-557b10492a29","Type":"ContainerDied","Data":"bdba8ec704bf8473641c973140864860d5a847718fdf0619ada765eec7c110af"}
Jul 07 02:29:23.642160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.642037    2561 scope.go:117] "RemoveContainer" containerID="44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"
Jul 07 02:29:23.642240 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.642217    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"
Jul 07 02:29:23.652341 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.652321    2561 scope.go:117] "RemoveContainer" containerID="f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"
Jul 07 02:29:23.664241 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.664217    2561 scope.go:117] "RemoveContainer" containerID="18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"
Jul 07 02:29:23.665057 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.665034    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:29:23.671101 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.671077    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-vbstl"]
Jul 07 02:29:23.673191 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.673177    2561 scope.go:117] "RemoveContainer" containerID="44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"
Jul 07 02:29:23.673486 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:23.673468    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3\": container with ID starting with 44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3 not found: ID does not exist" containerID="44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"
Jul 07 02:29:23.673558 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.673500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3"} err="failed to get container status \"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3\": rpc error: code = NotFound desc = could not find container \"44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3\": container with ID starting with 44c0c1813d066a1ed76e1d5ffd7f0c8ba3f3e19530d3f4d2ec641d30b99f82c3 not found: ID does not exist"
Jul 07 02:29:23.673558 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.673530    2561 scope.go:117] "RemoveContainer" containerID="f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"
Jul 07 02:29:23.673815 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:23.673798    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6\": container with ID starting with f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6 not found: ID does not exist" containerID="f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"
Jul 07 02:29:23.673853 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.673824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6"} err="failed to get container status \"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6\": rpc error: code = NotFound desc = could not find container \"f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6\": container with ID starting with f4fa3b375de06d128399feb0219d36bead06effdb9f8f02647b65d72454f16a6 not found: ID does not exist"
Jul 07 02:29:23.673853 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.673841    2561 scope.go:117] "RemoveContainer" containerID="18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"
Jul 07 02:29:23.674070 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:23.674053    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7\": container with ID starting with 18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7 not found: ID does not exist" containerID="18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"
Jul 07 02:29:23.674118 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.674075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7"} err="failed to get container status \"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7\": rpc error: code = NotFound desc = could not find container \"18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7\": container with ID starting with 18be512e20e66453e3486c6bfd30d781eb7fe72f75862e4a51bf1be45da4e7c7 not found: ID does not exist"
Jul 07 02:29:23.959976 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:23.959951    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:29:24.052377 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5j55\" (UniqueName: \"kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55\") pod \"e17e9087-204f-4750-9bf3-762232f66d66\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") "
Jul 07 02:29:24.052875 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052411    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config\") pod \"e17e9087-204f-4750-9bf3-762232f66d66\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") "
Jul 07 02:29:24.052875 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052494    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location\") pod \"e17e9087-204f-4750-9bf3-762232f66d66\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") "
Jul 07 02:29:24.052875 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052535    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert\") pod \"e17e9087-204f-4750-9bf3-762232f66d66\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") "
Jul 07 02:29:24.052875 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052571    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") pod \"e17e9087-204f-4750-9bf3-762232f66d66\" (UID: \"e17e9087-204f-4750-9bf3-762232f66d66\") "
Jul 07 02:29:24.053095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052949    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e17e9087-204f-4750-9bf3-762232f66d66" (UID: "e17e9087-204f-4750-9bf3-762232f66d66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:29:24.053095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.052977    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e17e9087-204f-4750-9bf3-762232f66d66" (UID: "e17e9087-204f-4750-9bf3-762232f66d66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:24.053095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.053055    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e17e9087-204f-4750-9bf3-762232f66d66" (UID: "e17e9087-204f-4750-9bf3-762232f66d66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:24.054710 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.054686    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55" (OuterVolumeSpecName: "kube-api-access-f5j55") pod "e17e9087-204f-4750-9bf3-762232f66d66" (UID: "e17e9087-204f-4750-9bf3-762232f66d66"). InnerVolumeSpecName "kube-api-access-f5j55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:29:24.054969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.054950    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e17e9087-204f-4750-9bf3-762232f66d66" (UID: "e17e9087-204f-4750-9bf3-762232f66d66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:29:24.154089 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.154063    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e17e9087-204f-4750-9bf3-762232f66d66-kserve-provision-location\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.154224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.154092    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-cabundle-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.154224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.154102    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e17e9087-204f-4750-9bf3-762232f66d66-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.154224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.154110    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5j55\" (UniqueName: \"kubernetes.io/projected/e17e9087-204f-4750-9bf3-762232f66d66-kube-api-access-f5j55\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.154224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.154118    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e17e9087-204f-4750-9bf3-762232f66d66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.651248 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.651215    2561 generic.go:356] "Generic (PLEG): container finished" podID="e17e9087-204f-4750-9bf3-762232f66d66" containerID="8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f" exitCode=0
Jul 07 02:29:24.651442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.651297    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"
Jul 07 02:29:24.651442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.651299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerDied","Data":"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"}
Jul 07 02:29:24.651442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.651380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx" event={"ID":"e17e9087-204f-4750-9bf3-762232f66d66","Type":"ContainerDied","Data":"5332929aff0fbd3c5a8cba7553d93d83aec731e0ea3c66e1f36bf9d9590325f9"}
Jul 07 02:29:24.651442 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.651422    2561 scope.go:117] "RemoveContainer" containerID="c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"
Jul 07 02:29:24.653856 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.653829    2561 generic.go:356] "Generic (PLEG): container finished" podID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerID="230da3a852eeaa45c1ddb1c106281941b9a21f641766221c9da12a0acc154ebd" exitCode=0
Jul 07 02:29:24.653990 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.653900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerDied","Data":"230da3a852eeaa45c1ddb1c106281941b9a21f641766221c9da12a0acc154ebd"}
Jul 07 02:29:24.660760 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.660737    2561 scope.go:117] "RemoveContainer" containerID="8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"
Jul 07 02:29:24.669921 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.669902    2561 scope.go:117] "RemoveContainer" containerID="97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"
Jul 07 02:29:24.677020 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.676998    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:29:24.678485 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.678467    2561 scope.go:117] "RemoveContainer" containerID="c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"
Jul 07 02:29:24.678903 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:24.678866    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4\": container with ID starting with c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4 not found: ID does not exist" containerID="c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"
Jul 07 02:29:24.679003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.678913    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4"} err="failed to get container status \"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4\": rpc error: code = NotFound desc = could not find container \"c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4\": container with ID starting with c60b43af01734d9ec7f843a2ba45a5201f72309e7a5cac244b4219bce0ed5fe4 not found: ID does not exist"
Jul 07 02:29:24.679003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.678963    2561 scope.go:117] "RemoveContainer" containerID="8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"
Jul 07 02:29:24.679375 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:24.679265    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f\": container with ID starting with 8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f not found: ID does not exist" containerID="8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"
Jul 07 02:29:24.679375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.679294    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f"} err="failed to get container status \"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f\": rpc error: code = NotFound desc = could not find container \"8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f\": container with ID starting with 8b8fb5d5d2784f3b8696bc22c02a31cf3206c1e7b2c453caeabbd3dea9149d8f not found: ID does not exist"
Jul 07 02:29:24.679375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.679316    2561 scope.go:117] "RemoveContainer" containerID="97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"
Jul 07 02:29:24.679890 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:29:24.679848    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3\": container with ID starting with 97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3 not found: ID does not exist" containerID="97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"
Jul 07 02:29:24.679987 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.679877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3"} err="failed to get container status \"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3\": rpc error: code = NotFound desc = could not find container \"97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3\": container with ID starting with 97dbd8d8eca4a6323e050cfd5ec74a080ded1a0e378b73ec4ce8a37cab6ca4f3 not found: ID does not exist"
Jul 07 02:29:24.682776 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.682756    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-658dd9c69f-csnfx"]
Jul 07 02:29:24.709762 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.709740    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:29:24.759309 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") pod \"2635b90c-7986-46a8-a45a-1a209eb475fb\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") "
Jul 07 02:29:24.759487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8\") pod \"2635b90c-7986-46a8-a45a-1a209eb475fb\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") "
Jul 07 02:29:24.759487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location\") pod \"2635b90c-7986-46a8-a45a-1a209eb475fb\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") "
Jul 07 02:29:24.759487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config\") pod \"2635b90c-7986-46a8-a45a-1a209eb475fb\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") "
Jul 07 02:29:24.759487 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert\") pod \"2635b90c-7986-46a8-a45a-1a209eb475fb\" (UID: \"2635b90c-7986-46a8-a45a-1a209eb475fb\") "
Jul 07 02:29:24.759803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759776    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2635b90c-7986-46a8-a45a-1a209eb475fb" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:29:24.759891 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759869    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2635b90c-7986-46a8-a45a-1a209eb475fb" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:24.759937 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.759869    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2635b90c-7986-46a8-a45a-1a209eb475fb" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:24.761617 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.761595    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2635b90c-7986-46a8-a45a-1a209eb475fb" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:29:24.761691 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.761598    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8" (OuterVolumeSpecName: "kube-api-access-slwl8") pod "2635b90c-7986-46a8-a45a-1a209eb475fb" (UID: "2635b90c-7986-46a8-a45a-1a209eb475fb"). InnerVolumeSpecName "kube-api-access-slwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:29:24.860283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.860234    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635b90c-7986-46a8-a45a-1a209eb475fb-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.860283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.860274    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-api-access-slwl8\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.860283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.860287    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635b90c-7986-46a8-a45a-1a209eb475fb-kserve-provision-location\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.860550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.860299    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:24.860550 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:24.860313    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635b90c-7986-46a8-a45a-1a209eb475fb-cabundle-cert\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:29:25.283898 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.283809    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="636ca02f-d3be-454e-81ab-557b10492a29" path="/var/lib/kubelet/pods/636ca02f-d3be-454e-81ab-557b10492a29/volumes"
Jul 07 02:29:25.284324 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.284310    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17e9087-204f-4750-9bf3-762232f66d66" path="/var/lib/kubelet/pods/e17e9087-204f-4750-9bf3-762232f66d66/volumes"
Jul 07 02:29:25.412015 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.411977    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:29:25.660840 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.660803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn" event={"ID":"2635b90c-7986-46a8-a45a-1a209eb475fb","Type":"ContainerDied","Data":"c616b3ff257c59868c990b81db5e14b30d22b369070704a6d11f826eedeb3f5d"}
Jul 07 02:29:25.660840 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.660822    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"
Jul 07 02:29:25.661132 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.660881    2561 scope.go:117] "RemoveContainer" containerID="dc50ec0c61a6f795e70b3449f2ff35fb6f93ce2f04df22393b71cfeb6fe3a830"
Jul 07 02:29:25.669952 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.669930    2561 scope.go:117] "RemoveContainer" containerID="230da3a852eeaa45c1ddb1c106281941b9a21f641766221c9da12a0acc154ebd"
Jul 07 02:29:25.678922 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.678900    2561 scope.go:117] "RemoveContainer" containerID="4cbb7bc0ca6457a7c973afaad9f7da4e3ab76f4ce4cb42b3110a9c7354bf82b5"
Jul 07 02:29:25.680584 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.680565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:29:25.686481 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:25.686456    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-ccccddb65-rwvgn"]
Jul 07 02:29:26.635244 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:26.635217    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:29:26.635785 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:26.635758    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:29:27.284229 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:27.284197    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" path="/var/lib/kubelet/pods/2635b90c-7986-46a8-a45a-1a209eb475fb/volumes"
Jul 07 02:29:35.412635 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:35.412600    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:29:36.636005 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:36.635966    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:29:46.636495 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:46.636450    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:29:56.635919 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:29:56.635868    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:30:06.636736 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:30:06.636649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:33:09.195221 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:33:09.195121    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:33:09.195800 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:33:09.195394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:38:03.518584 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.518546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:38:03.521212 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.518866    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" containerID="cri-o://9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935" gracePeriod=30
Jul 07 02:38:03.521212 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.518893    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kube-rbac-proxy" containerID="cri-o://02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83" gracePeriod=30
Jul 07 02:38:03.627058 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627025    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:38:03.627490 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627461    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="storage-initializer"
Jul 07 02:38:03.627490 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627481    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="storage-initializer"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627496    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="storage-initializer"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="storage-initializer"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="storage-initializer"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627533    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="storage-initializer"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627559    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627575    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kube-rbac-proxy"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kube-rbac-proxy"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627603    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627614    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627638    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kube-rbac-proxy"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kube-rbac-proxy"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627672    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kube-rbac-proxy"
Jul 07 02:38:03.627683 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627680    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kube-rbac-proxy"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627765    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kube-rbac-proxy"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627778    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kserve-container"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627788    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="636ca02f-d3be-454e-81ab-557b10492a29" containerName="kube-rbac-proxy"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627799    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2635b90c-7986-46a8-a45a-1a209eb475fb" containerName="kserve-container"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627807    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kserve-container"
Jul 07 02:38:03.628438 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.627817    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17e9087-204f-4750-9bf3-762232f66d66" containerName="kube-rbac-proxy"
Jul 07 02:38:03.631338 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.631315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.633911 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.633875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cec47-predictor-serving-cert\""
Jul 07 02:38:03.634063 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.633875    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cec47-kube-rbac-proxy-sar-config\""
Jul 07 02:38:03.641892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.641820    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:38:03.743197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.743157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsfd\" (UniqueName: \"kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.743197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.743200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.743411 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.743294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.843800 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.843766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsfd\" (UniqueName: \"kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.843800 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.843804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.844076 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.843843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.844076 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:03.843940    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cec47-predictor-serving-cert: secret "success-200-isvc-cec47-predictor-serving-cert" not found
Jul 07 02:38:03.844076 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:03.843999    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls podName:42e934da-0017-49bf-b2af-90c5da5e4ff5 nodeName:}" failed. No retries permitted until 2026-07-07 02:38:04.34398499 +0000 UTC m=+1195.651303357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls") pod "success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" (UID: "42e934da-0017-49bf-b2af-90c5da5e4ff5") : secret "success-200-isvc-cec47-predictor-serving-cert" not found
Jul 07 02:38:03.844478 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.844458    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.852606 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.852578    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsfd\" (UniqueName: \"kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:03.935061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.935022    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerID="02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83" exitCode=2
Jul 07 02:38:03.935061 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:03.935066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerDied","Data":"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"}
Jul 07 02:38:04.348128 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.348097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:04.350608 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.350577    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") pod \"success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") " pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:04.545434 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.545398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:04.677748 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.677725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:38:04.680175 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:38:04.680146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e934da_0017_49bf_b2af_90c5da5e4ff5.slice/crio-8bc02f2dc024f225bfd0d68306ce1271fbe6da17c9aa16129c12c8af2587e012 WatchSource:0}: Error finding container 8bc02f2dc024f225bfd0d68306ce1271fbe6da17c9aa16129c12c8af2587e012: Status 404 returned error can't find the container with id 8bc02f2dc024f225bfd0d68306ce1271fbe6da17c9aa16129c12c8af2587e012
Jul 07 02:38:04.942529 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.942418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerStarted","Data":"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"}
Jul 07 02:38:04.942698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.942564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:04.942698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.942601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerStarted","Data":"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"}
Jul 07 02:38:04.942698 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.942657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerStarted","Data":"8bc02f2dc024f225bfd0d68306ce1271fbe6da17c9aa16129c12c8af2587e012"}
Jul 07 02:38:04.965334 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:04.965224    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podStartSLOduration=1.96520591 podStartE2EDuration="1.96520591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:04.964874451 +0000 UTC m=+1196.272192843" watchObservedRunningTime="2026-07-07 02:38:04.96520591 +0000 UTC m=+1196.272524301"
Jul 07 02:38:05.407174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:05.407133    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 02:38:05.411472 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:05.411438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 02:38:05.947287 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:05.947258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:05.948697 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:05.948667    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:06.770832 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.770808    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:38:06.871633 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.871592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r\") pod \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") "
Jul 07 02:38:06.871828 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.871690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config\") pod \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") "
Jul 07 02:38:06.871828 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.871727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls\") pod \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\" (UID: \"bd23a8eb-edd5-4d82-9523-36e7aff2e825\") "
Jul 07 02:38:06.872047 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.872012    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd23a8eb-edd5-4d82-9523-36e7aff2e825" (UID: "bd23a8eb-edd5-4d82-9523-36e7aff2e825"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:38:06.873852 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.873829    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r" (OuterVolumeSpecName: "kube-api-access-ttg2r") pod "bd23a8eb-edd5-4d82-9523-36e7aff2e825" (UID: "bd23a8eb-edd5-4d82-9523-36e7aff2e825"). InnerVolumeSpecName "kube-api-access-ttg2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:38:06.873852 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.873839    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd23a8eb-edd5-4d82-9523-36e7aff2e825" (UID: "bd23a8eb-edd5-4d82-9523-36e7aff2e825"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:38:06.952814 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.952782    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerID="9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935" exitCode=0
Jul 07 02:38:06.953291 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.952827    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"
Jul 07 02:38:06.953291 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.952875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerDied","Data":"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"}
Jul 07 02:38:06.953291 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.952949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7" event={"ID":"bd23a8eb-edd5-4d82-9523-36e7aff2e825","Type":"ContainerDied","Data":"b8c6602ceacc4df1bb12baa07e98802b2d09a85620e7f0b0382f1803e3606b83"}
Jul 07 02:38:06.953291 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.952991    2561 scope.go:117] "RemoveContainer" containerID="02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"
Jul 07 02:38:06.953501 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.953467    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:06.961784 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.961758    2561 scope.go:117] "RemoveContainer" containerID="9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"
Jul 07 02:38:06.969177 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.969161    2561 scope.go:117] "RemoveContainer" containerID="02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"
Jul 07 02:38:06.969440 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:06.969422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83\": container with ID starting with 02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83 not found: ID does not exist" containerID="02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"
Jul 07 02:38:06.969489 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.969449    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83"} err="failed to get container status \"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83\": rpc error: code = NotFound desc = could not find container \"02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83\": container with ID starting with 02e6cda26f81fbf568d22c282c0aa7bf93feec91ae08f0e13a0caba30bfb1a83 not found: ID does not exist"
Jul 07 02:38:06.969489 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.969466    2561 scope.go:117] "RemoveContainer" containerID="9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"
Jul 07 02:38:06.969745 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:06.969727    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935\": container with ID starting with 9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935 not found: ID does not exist" containerID="9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"
Jul 07 02:38:06.969803 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.969751    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935"} err="failed to get container status \"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935\": rpc error: code = NotFound desc = could not find container \"9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935\": container with ID starting with 9d546d69e64b764d6416aba2c4c5c9827fbb52a25eafa7026f344117f9bde935 not found: ID does not exist"
Jul 07 02:38:06.973243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.973219    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-api-access-ttg2r\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:06.973327 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.973248    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd23a8eb-edd5-4d82-9523-36e7aff2e825-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:06.973327 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.973263    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd23a8eb-edd5-4d82-9523-36e7aff2e825-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:06.974855 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.974829    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:38:06.978301 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:06.978282    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3ea7f-predictor-7f4997b577-vs7b7"]
Jul 07 02:38:07.284262 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:07.284166    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" path="/var/lib/kubelet/pods/bd23a8eb-edd5-4d82-9523-36e7aff2e825/volumes"
Jul 07 02:38:09.224687 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:09.224659    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:38:09.225777 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:09.225756    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:38:11.957853 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:11.957825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:11.958294 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:11.958266    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:21.959254 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:21.959212    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:31.958960 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:31.958921    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:33.509081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.509049    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:38:33.509546 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.509316    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" containerID="cri-o://4000c9e21c652e36dda27d8962a1de416118146c5e373b460014b5d53936d7a5" gracePeriod=30
Jul 07 02:38:33.509546 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.509372    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kube-rbac-proxy" containerID="cri-o://7fe17ea39038924f77ba89ff7f5f1ab0415179ce71af96dc561ffcd29afe7c1f" gracePeriod=30
Jul 07 02:38:33.586262 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586226    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:38:33.586662 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586649    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container"
Jul 07 02:38:33.586721 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586663    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container"
Jul 07 02:38:33.586721 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586680    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kube-rbac-proxy"
Jul 07 02:38:33.586721 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586685    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kube-rbac-proxy"
Jul 07 02:38:33.586828 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586765    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kserve-container"
Jul 07 02:38:33.586828 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.586776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd23a8eb-edd5-4d82-9523-36e7aff2e825" containerName="kube-rbac-proxy"
Jul 07 02:38:33.598269 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.598246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.602092 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.602070    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-524cd-kube-rbac-proxy-sar-config\""
Jul 07 02:38:33.602220 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.602099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-524cd-predictor-serving-cert\""
Jul 07 02:38:33.609290 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.609266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddns\" (UniqueName: \"kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.609393 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.609310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.609449 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.609421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.612283 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.612262    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:38:33.710205 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.710172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddns\" (UniqueName: \"kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.710205 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.710213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.710468 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.710295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.710468 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:33.710392    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-524cd-predictor-serving-cert: secret "success-200-isvc-524cd-predictor-serving-cert" not found
Jul 07 02:38:33.710468 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:38:33.710447    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls podName:12539b23-ceb1-41d4-97d9-732b3316d37e nodeName:}" failed. No retries permitted until 2026-07-07 02:38:34.210427485 +0000 UTC m=+1225.517745853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls") pod "success-200-isvc-524cd-predictor-b954c6b45-pqbdm" (UID: "12539b23-ceb1-41d4-97d9-732b3316d37e") : secret "success-200-isvc-524cd-predictor-serving-cert" not found
Jul 07 02:38:33.711167 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.711138    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:33.726243 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:33.726219    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddns\" (UniqueName: \"kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:34.077819 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.077787    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerID="7fe17ea39038924f77ba89ff7f5f1ab0415179ce71af96dc561ffcd29afe7c1f" exitCode=2
Jul 07 02:38:34.077819 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.077822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerDied","Data":"7fe17ea39038924f77ba89ff7f5f1ab0415179ce71af96dc561ffcd29afe7c1f"}
Jul 07 02:38:34.215344 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.215310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:34.217782 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.217764    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") pod \"success-200-isvc-524cd-predictor-b954c6b45-pqbdm\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") " pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:34.508879 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.508784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:34.650323 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:34.650300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:38:34.652764 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:38:34.652740    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12539b23_ceb1_41d4_97d9_732b3316d37e.slice/crio-aa74007f08c6ea3337fafa9b87bea2143f323c48b202d1ae390ecd77cbfd0419 WatchSource:0}: Error finding container aa74007f08c6ea3337fafa9b87bea2143f323c48b202d1ae390ecd77cbfd0419: Status 404 returned error can't find the container with id aa74007f08c6ea3337fafa9b87bea2143f323c48b202d1ae390ecd77cbfd0419
Jul 07 02:38:35.085275 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:35.085231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerStarted","Data":"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"}
Jul 07 02:38:35.085461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:35.085313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerStarted","Data":"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"}
Jul 07 02:38:35.085461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:35.085351    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:35.085461 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:35.085379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerStarted","Data":"aa74007f08c6ea3337fafa9b87bea2143f323c48b202d1ae390ecd77cbfd0419"}
Jul 07 02:38:35.103099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:35.102999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podStartSLOduration=2.102976879 podStartE2EDuration="2.102976879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:35.102888855 +0000 UTC m=+1226.410207271" watchObservedRunningTime="2026-07-07 02:38:35.102976879 +0000 UTC m=+1226.410295272"
Jul 07 02:38:36.089682 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:36.089647    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:36.090990 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:36.090965    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:38:36.629639 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:36.629579    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 07 02:38:36.636224 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:36.636166    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 02:38:37.097741 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.097713    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerID="4000c9e21c652e36dda27d8962a1de416118146c5e373b460014b5d53936d7a5" exitCode=0
Jul 07 02:38:37.098120 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.097796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerDied","Data":"4000c9e21c652e36dda27d8962a1de416118146c5e373b460014b5d53936d7a5"}
Jul 07 02:38:37.098175 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.098129    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:38:37.165535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.165508    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:38:37.241737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.241647    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzccx\" (UniqueName: \"kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx\") pod \"9c913aa5-b07f-433f-8f22-f15f154b5c35\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") "
Jul 07 02:38:37.241737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.241704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config\") pod \"9c913aa5-b07f-433f-8f22-f15f154b5c35\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") "
Jul 07 02:38:37.242037 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.241843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") pod \"9c913aa5-b07f-433f-8f22-f15f154b5c35\" (UID: \"9c913aa5-b07f-433f-8f22-f15f154b5c35\") "
Jul 07 02:38:37.242095 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.242078    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c913aa5-b07f-433f-8f22-f15f154b5c35" (UID: "9c913aa5-b07f-433f-8f22-f15f154b5c35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:38:37.244655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.244590    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c913aa5-b07f-433f-8f22-f15f154b5c35" (UID: "9c913aa5-b07f-433f-8f22-f15f154b5c35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:38:37.244655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.244645    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx" (OuterVolumeSpecName: "kube-api-access-xzccx") pod "9c913aa5-b07f-433f-8f22-f15f154b5c35" (UID: "9c913aa5-b07f-433f-8f22-f15f154b5c35"). InnerVolumeSpecName "kube-api-access-xzccx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:38:37.343278 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.343140    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzccx\" (UniqueName: \"kubernetes.io/projected/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-api-access-xzccx\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:37.343278 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.343169    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c913aa5-b07f-433f-8f22-f15f154b5c35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:37.343278 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:37.343183    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c913aa5-b07f-433f-8f22-f15f154b5c35-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:38:38.104040 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.104010    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"
Jul 07 02:38:38.104432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.104005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t" event={"ID":"9c913aa5-b07f-433f-8f22-f15f154b5c35","Type":"ContainerDied","Data":"280900409d091ace93bae461aae8c606a907587c845571dc09963eee270fb8c6"}
Jul 07 02:38:38.104432 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.104180    2561 scope.go:117] "RemoveContainer" containerID="7fe17ea39038924f77ba89ff7f5f1ab0415179ce71af96dc561ffcd29afe7c1f"
Jul 07 02:38:38.112955 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.112932    2561 scope.go:117] "RemoveContainer" containerID="4000c9e21c652e36dda27d8962a1de416118146c5e373b460014b5d53936d7a5"
Jul 07 02:38:38.128324 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.128298    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:38:38.129922 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:38.129902    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-821ea-predictor-6597d66d67-9vj2t"]
Jul 07 02:38:39.283881 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:39.283841    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" path="/var/lib/kubelet/pods/9c913aa5-b07f-433f-8f22-f15f154b5c35/volumes"
Jul 07 02:38:41.958979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:41.958926    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 02:38:42.103677 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:42.103617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:38:42.104142 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:42.104117    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:38:51.959355 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:51.959328    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:38:52.105161 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:38:52.105120    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:39:02.104796 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:02.104698    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:39:12.104199 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:12.104157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 02:39:13.880526 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.880484    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:39:13.880997 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.880766    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container" containerID="cri-o://fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0" gracePeriod=30
Jul 07 02:39:13.880997 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.880848    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kube-rbac-proxy" containerID="cri-o://fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855" gracePeriod=30
Jul 07 02:39:13.930655 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.930604    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:39:13.931039 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931026    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kube-rbac-proxy"
Jul 07 02:39:13.931093 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931043    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kube-rbac-proxy"
Jul 07 02:39:13.931093 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931073    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container"
Jul 07 02:39:13.931093 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931079    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container"
Jul 07 02:39:13.931203 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931136    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kserve-container"
Jul 07 02:39:13.931203 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.931151    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c913aa5-b07f-433f-8f22-f15f154b5c35" containerName="kube-rbac-proxy"
Jul 07 02:39:13.935576 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.935556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:13.937930 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.937908    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4a8c-predictor-serving-cert\""
Jul 07 02:39:13.938043 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.937910    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4a8c-kube-rbac-proxy-sar-config\""
Jul 07 02:39:13.943003 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.942976    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:39:13.980921 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.980886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:13.981091 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.980936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:13.981091 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:13.981058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.081999 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.081958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.082181 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.082035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.082181 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.082061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.082181 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:14.082163    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-serving-cert: secret "success-200-isvc-e4a8c-predictor-serving-cert" not found
Jul 07 02:39:14.082304 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:14.082226    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls podName:d21bec98-849a-41f5-8b49-622f9f6f0ea7 nodeName:}" failed. No retries permitted until 2026-07-07 02:39:14.582211259 +0000 UTC m=+1265.889529627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls") pod "success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" (UID: "d21bec98-849a-41f5-8b49-622f9f6f0ea7") : secret "success-200-isvc-e4a8c-predictor-serving-cert" not found
Jul 07 02:39:14.082680 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.082663    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.090383 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.090354    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.275795 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.275700    2561 generic.go:356] "Generic (PLEG): container finished" podID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerID="fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855" exitCode=2
Jul 07 02:39:14.275795 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.275774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerDied","Data":"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"}
Jul 07 02:39:14.586098 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.586061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.588670 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.588645    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") pod \"success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.847932 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.847839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:14.987422 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:14.987389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:39:15.283789 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:15.283756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerStarted","Data":"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"}
Jul 07 02:39:15.283960 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:15.283818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:15.283960 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:15.283838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerStarted","Data":"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"}
Jul 07 02:39:15.283960 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:15.283856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerStarted","Data":"f78587893b13f7de6bc49f8e4773b95102dd0e23f2c03c8f2375b707e0f63981"}
Jul 07 02:39:15.300870 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:15.300818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podStartSLOduration=2.3008010309999998 podStartE2EDuration="2.300801031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:39:15.298754052 +0000 UTC m=+1266.606072456" watchObservedRunningTime="2026-07-07 02:39:15.300801031 +0000 UTC m=+1266.608119420"
Jul 07 02:39:16.285922 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:16.285892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:16.287209 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:16.287181    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:17.029657 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.029612    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:39:17.109326 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.109234    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxsfd\" (UniqueName: \"kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd\") pod \"42e934da-0017-49bf-b2af-90c5da5e4ff5\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") "
Jul 07 02:39:17.109326 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.109294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") pod \"42e934da-0017-49bf-b2af-90c5da5e4ff5\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") "
Jul 07 02:39:17.109506 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.109363    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config\") pod \"42e934da-0017-49bf-b2af-90c5da5e4ff5\" (UID: \"42e934da-0017-49bf-b2af-90c5da5e4ff5\") "
Jul 07 02:39:17.109784 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.109758    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42e934da-0017-49bf-b2af-90c5da5e4ff5" (UID: "42e934da-0017-49bf-b2af-90c5da5e4ff5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:39:17.111526 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.111498    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42e934da-0017-49bf-b2af-90c5da5e4ff5" (UID: "42e934da-0017-49bf-b2af-90c5da5e4ff5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:39:17.111648 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.111587    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd" (OuterVolumeSpecName: "kube-api-access-cxsfd") pod "42e934da-0017-49bf-b2af-90c5da5e4ff5" (UID: "42e934da-0017-49bf-b2af-90c5da5e4ff5"). InnerVolumeSpecName "kube-api-access-cxsfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:39:17.211001 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.210967    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:17.211001 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.210995    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsfd\" (UniqueName: \"kubernetes.io/projected/42e934da-0017-49bf-b2af-90c5da5e4ff5-kube-api-access-cxsfd\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:17.211001 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.211004    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42e934da-0017-49bf-b2af-90c5da5e4ff5-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:17.292124 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292091    2561 generic.go:356] "Generic (PLEG): container finished" podID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerID="fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0" exitCode=0
Jul 07 02:39:17.292516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292159    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"
Jul 07 02:39:17.292516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerDied","Data":"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"}
Jul 07 02:39:17.292516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" event={"ID":"42e934da-0017-49bf-b2af-90c5da5e4ff5","Type":"ContainerDied","Data":"8bc02f2dc024f225bfd0d68306ce1271fbe6da17c9aa16129c12c8af2587e012"}
Jul 07 02:39:17.292516 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292257    2561 scope.go:117] "RemoveContainer" containerID="fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"
Jul 07 02:39:17.292871 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.292846    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:17.300813 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.300792    2561 scope.go:117] "RemoveContainer" containerID="fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"
Jul 07 02:39:17.308718 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.308699    2561 scope.go:117] "RemoveContainer" containerID="fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"
Jul 07 02:39:17.308981 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:17.308964    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855\": container with ID starting with fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855 not found: ID does not exist" containerID="fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"
Jul 07 02:39:17.309041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.308991    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855"} err="failed to get container status \"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855\": rpc error: code = NotFound desc = could not find container \"fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855\": container with ID starting with fec2a3979bf63f65828824f9238853ca7093a05693ada53fba64517e0c0db855 not found: ID does not exist"
Jul 07 02:39:17.309041 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.309006    2561 scope.go:117] "RemoveContainer" containerID="fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"
Jul 07 02:39:17.309247 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:17.309231    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0\": container with ID starting with fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0 not found: ID does not exist" containerID="fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"
Jul 07 02:39:17.309296 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.309251    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0"} err="failed to get container status \"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0\": rpc error: code = NotFound desc = could not find container \"fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0\": container with ID starting with fa74c9eb90298bf0a140e52d37251aa21c117d713e8418b0f9dce36810c22df0 not found: ID does not exist"
Jul 07 02:39:17.313456 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.313424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:39:17.317870 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.317842    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb"]
Jul 07 02:39:17.954551 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:17.954504    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cec47-predictor-58dfbcbf4f-h6hkb" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 07 02:39:19.283661 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:19.283606    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" path="/var/lib/kubelet/pods/42e934da-0017-49bf-b2af-90c5da5e4ff5/volumes"
Jul 07 02:39:22.105737 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:22.105708    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:39:22.297497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:22.297464    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:39:22.297977 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:22.297955    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:32.297892 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:32.297851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:42.297913 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:42.297873    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:43.784294 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.784258    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:39:43.784746 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.784615    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container" containerID="cri-o://49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0" gracePeriod=30
Jul 07 02:39:43.784746 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.784664    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kube-rbac-proxy" containerID="cri-o://b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37" gracePeriod=30
Jul 07 02:39:43.868734 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.868697    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:39:43.869070 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869059    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kube-rbac-proxy"
Jul 07 02:39:43.869114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869072    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kube-rbac-proxy"
Jul 07 02:39:43.869114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869096    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container"
Jul 07 02:39:43.869114 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869101    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container"
Jul 07 02:39:43.869210 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869154    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kserve-container"
Jul 07 02:39:43.869210 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.869165    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e934da-0017-49bf-b2af-90c5da5e4ff5" containerName="kube-rbac-proxy"
Jul 07 02:39:43.872342 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.872326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:43.875009 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.874989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c238-predictor-serving-cert\""
Jul 07 02:39:43.875091 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.874990    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c238-kube-rbac-proxy-sar-config\""
Jul 07 02:39:43.882160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.882138    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:39:43.955011 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.954975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:43.955198 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.955021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:43.955198 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:43.955112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9cx\" (UniqueName: \"kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.056055 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.055972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.056055 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.056013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.056268 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.056068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9cx\" (UniqueName: \"kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.056268 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:44.056222    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7c238-predictor-serving-cert: secret "success-200-isvc-7c238-predictor-serving-cert" not found
Jul 07 02:39:44.056342 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:44.056301    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls podName:e367f8d4-340b-4341-b0af-8df7a4f80c32 nodeName:}" failed. No retries permitted until 2026-07-07 02:39:44.556283087 +0000 UTC m=+1295.863601460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls") pod "success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" (UID: "e367f8d4-340b-4341-b0af-8df7a4f80c32") : secret "success-200-isvc-7c238-predictor-serving-cert" not found
Jul 07 02:39:44.056694 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.056678    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.065329 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.065302    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9cx\" (UniqueName: \"kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.412600 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.412564    2561 generic.go:356] "Generic (PLEG): container finished" podID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerID="b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37" exitCode=2
Jul 07 02:39:44.412772 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.412655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerDied","Data":"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"}
Jul 07 02:39:44.561214 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.561168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.563719 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.563699    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") pod \"success-200-isvc-7c238-predictor-54d4799ddc-wjsvv\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") " pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.783723 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.783600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:44.921468 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:44.921439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:39:44.923551 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:39:44.923520    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode367f8d4_340b_4341_b0af_8df7a4f80c32.slice/crio-8990d1a74cc3993f41fbc96fba6426216d73da3d75500ab8fa2d10c2b526fd25 WatchSource:0}: Error finding container 8990d1a74cc3993f41fbc96fba6426216d73da3d75500ab8fa2d10c2b526fd25: Status 404 returned error can't find the container with id 8990d1a74cc3993f41fbc96fba6426216d73da3d75500ab8fa2d10c2b526fd25
Jul 07 02:39:45.419276 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:45.419244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerStarted","Data":"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"}
Jul 07 02:39:45.419462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:45.419306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerStarted","Data":"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"}
Jul 07 02:39:45.419462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:45.419329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerStarted","Data":"8990d1a74cc3993f41fbc96fba6426216d73da3d75500ab8fa2d10c2b526fd25"}
Jul 07 02:39:45.419462 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:45.419373    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:45.437408 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:45.437346    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podStartSLOduration=2.437323286 podStartE2EDuration="2.437323286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:39:45.436901135 +0000 UTC m=+1296.744219526" watchObservedRunningTime="2026-07-07 02:39:45.437323286 +0000 UTC m=+1296.744641677"
Jul 07 02:39:46.424551 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:46.424518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:46.425891 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:46.425864    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:39:47.098743 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.098702    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 07 02:39:47.348037 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.348011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:39:47.430512 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.430412    2561 generic.go:356] "Generic (PLEG): container finished" podID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerID="49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0" exitCode=0
Jul 07 02:39:47.430512 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.430465    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"
Jul 07 02:39:47.431073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.430518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerDied","Data":"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"}
Jul 07 02:39:47.431073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.430585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm" event={"ID":"12539b23-ceb1-41d4-97d9-732b3316d37e","Type":"ContainerDied","Data":"aa74007f08c6ea3337fafa9b87bea2143f323c48b202d1ae390ecd77cbfd0419"}
Jul 07 02:39:47.431073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.430643    2561 scope.go:117] "RemoveContainer" containerID="b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"
Jul 07 02:39:47.431073 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.431061    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:39:47.440595 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.440579    2561 scope.go:117] "RemoveContainer" containerID="49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"
Jul 07 02:39:47.448748 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.448720    2561 scope.go:117] "RemoveContainer" containerID="b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"
Jul 07 02:39:47.449042 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:47.449024    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37\": container with ID starting with b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37 not found: ID does not exist" containerID="b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"
Jul 07 02:39:47.449118 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.449051    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37"} err="failed to get container status \"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37\": rpc error: code = NotFound desc = could not find container \"b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37\": container with ID starting with b92e26648f19012500fe2836bf757bf99f144930968ede7d7f674744cc2b7c37 not found: ID does not exist"
Jul 07 02:39:47.449118 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.449069    2561 scope.go:117] "RemoveContainer" containerID="49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"
Jul 07 02:39:47.449292 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:39:47.449274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0\": container with ID starting with 49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0 not found: ID does not exist" containerID="49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"
Jul 07 02:39:47.449332 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.449301    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0"} err="failed to get container status \"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0\": rpc error: code = NotFound desc = could not find container \"49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0\": container with ID starting with 49562065e3ea58dd95b28d4dce59691af0f6a2c40a1b09e6e4775840965c25e0 not found: ID does not exist"
Jul 07 02:39:47.486686 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.486654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vddns\" (UniqueName: \"kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns\") pod \"12539b23-ceb1-41d4-97d9-732b3316d37e\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") "
Jul 07 02:39:47.486854 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.486774    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") pod \"12539b23-ceb1-41d4-97d9-732b3316d37e\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") "
Jul 07 02:39:47.486854 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.486840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config\") pod \"12539b23-ceb1-41d4-97d9-732b3316d37e\" (UID: \"12539b23-ceb1-41d4-97d9-732b3316d37e\") "
Jul 07 02:39:47.487232 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.487203    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12539b23-ceb1-41d4-97d9-732b3316d37e" (UID: "12539b23-ceb1-41d4-97d9-732b3316d37e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:39:47.489010 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.488990    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns" (OuterVolumeSpecName: "kube-api-access-vddns") pod "12539b23-ceb1-41d4-97d9-732b3316d37e" (UID: "12539b23-ceb1-41d4-97d9-732b3316d37e"). InnerVolumeSpecName "kube-api-access-vddns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:39:47.489115 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.489092    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12539b23-ceb1-41d4-97d9-732b3316d37e" (UID: "12539b23-ceb1-41d4-97d9-732b3316d37e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:39:47.588168 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.588136    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12539b23-ceb1-41d4-97d9-732b3316d37e-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:47.588168 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.588166    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:47.588168 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.588180    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddns\" (UniqueName: \"kubernetes.io/projected/12539b23-ceb1-41d4-97d9-732b3316d37e-kube-api-access-vddns\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:39:47.753860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.753824    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:39:47.758366 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:47.758338    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-524cd-predictor-b954c6b45-pqbdm"]
Jul 07 02:39:49.284983 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:49.284949    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" path="/var/lib/kubelet/pods/12539b23-ceb1-41d4-97d9-732b3316d37e/volumes"
Jul 07 02:39:52.298251 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:52.298207    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:39:52.435528 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:52.435499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:39:52.436071 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:39:52.436032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:40:02.298513 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:40:02.298479    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:40:02.436152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:40:02.436106    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:40:12.436349 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:40:12.436310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:40:22.436110 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:40:22.436070    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:40:32.436692 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:40:32.436660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:43:09.253956 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:43:09.253914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:43:09.256416 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:43:09.255917    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:48:09.283927 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:09.283821    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:48:09.289862 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:09.286285    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:48:28.871071 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.871034    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:48:28.871499 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.871305    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" containerID="cri-o://bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39" gracePeriod=30
Jul 07 02:48:28.871499 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.871336    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kube-rbac-proxy" containerID="cri-o://9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93" gracePeriod=30
Jul 07 02:48:28.955400 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955359    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:48:28.955790 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955776    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kube-rbac-proxy"
Jul 07 02:48:28.955868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955792    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kube-rbac-proxy"
Jul 07 02:48:28.955868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955804    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container"
Jul 07 02:48:28.955868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955809    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container"
Jul 07 02:48:28.955868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kube-rbac-proxy"
Jul 07 02:48:28.956081 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.955876    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="12539b23-ceb1-41d4-97d9-732b3316d37e" containerName="kserve-container"
Jul 07 02:48:28.959043 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.959019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:28.961603 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.961585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4831c-kube-rbac-proxy-sar-config\""
Jul 07 02:48:28.962146 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.962131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4831c-predictor-serving-cert\""
Jul 07 02:48:28.969531 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:28.969498    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:48:29.054758 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.054718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.054979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.054848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjp85\" (UniqueName: \"kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.054979 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.054930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.155498 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.155394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.155498 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.155456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjp85\" (UniqueName: \"kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.155498 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.155487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.156174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.156155    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.158064 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.158046    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.164217 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.164190    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjp85\" (UniqueName: \"kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85\") pod \"success-200-isvc-4831c-predictor-854cc4c887-64bp4\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") " pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.271653 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.271582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.418191 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.418096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:48:29.421065 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:48:29.421035    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75f06b3_7b59_4a23_9b2e_e4c08af9b39a.slice/crio-77b7b789ba34dbb0580f4b79813a663a8be5ac5cd3977b24fad2c54898e6cb3e WatchSource:0}: Error finding container 77b7b789ba34dbb0580f4b79813a663a8be5ac5cd3977b24fad2c54898e6cb3e: Status 404 returned error can't find the container with id 77b7b789ba34dbb0580f4b79813a663a8be5ac5cd3977b24fad2c54898e6cb3e
Jul 07 02:48:29.711761 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.711650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerStarted","Data":"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"}
Jul 07 02:48:29.711761 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.711722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerStarted","Data":"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"}
Jul 07 02:48:29.711761 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.711756    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.712086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.711782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerStarted","Data":"77b7b789ba34dbb0580f4b79813a663a8be5ac5cd3977b24fad2c54898e6cb3e"}
Jul 07 02:48:29.712086 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.711810    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:29.713556 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.713530    2561 generic.go:356] "Generic (PLEG): container finished" podID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerID="9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93" exitCode=2
Jul 07 02:48:29.713717 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.713587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerDied","Data":"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"}
Jul 07 02:48:29.713717 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.713681    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:48:29.730782 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:29.730735    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podStartSLOduration=1.730719854 podStartE2EDuration="1.730719854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:48:29.729139295 +0000 UTC m=+1821.036457685" watchObservedRunningTime="2026-07-07 02:48:29.730719854 +0000 UTC m=+1821.038038300"
Jul 07 02:48:30.718964 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:30.718921    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:48:32.293752 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.293704    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 07 02:48:32.298075 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.298050    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 07 02:48:32.529716 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.529686    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:48:32.586162 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.586121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") pod \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") "
Jul 07 02:48:32.586375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.586187    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22\") pod \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") "
Jul 07 02:48:32.586375 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.586257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config\") pod \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\" (UID: \"d21bec98-849a-41f5-8b49-622f9f6f0ea7\") "
Jul 07 02:48:32.586711 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.586683    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d21bec98-849a-41f5-8b49-622f9f6f0ea7" (UID: "d21bec98-849a-41f5-8b49-622f9f6f0ea7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:48:32.588453 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.588429    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d21bec98-849a-41f5-8b49-622f9f6f0ea7" (UID: "d21bec98-849a-41f5-8b49-622f9f6f0ea7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:48:32.588551 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.588533    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22" (OuterVolumeSpecName: "kube-api-access-xmb22") pod "d21bec98-849a-41f5-8b49-622f9f6f0ea7" (UID: "d21bec98-849a-41f5-8b49-622f9f6f0ea7"). InnerVolumeSpecName "kube-api-access-xmb22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:48:32.687197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.687154    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:48:32.687197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.687187    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21bec98-849a-41f5-8b49-622f9f6f0ea7-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:48:32.687197 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.687198    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/d21bec98-849a-41f5-8b49-622f9f6f0ea7-kube-api-access-xmb22\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:48:32.730069 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.730032    2561 generic.go:356] "Generic (PLEG): container finished" podID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerID="bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39" exitCode=0
Jul 07 02:48:32.730266 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.730101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"
Jul 07 02:48:32.730266 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.730125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerDied","Data":"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"}
Jul 07 02:48:32.730266 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.730189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf" event={"ID":"d21bec98-849a-41f5-8b49-622f9f6f0ea7","Type":"ContainerDied","Data":"f78587893b13f7de6bc49f8e4773b95102dd0e23f2c03c8f2375b707e0f63981"}
Jul 07 02:48:32.730266 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.730216    2561 scope.go:117] "RemoveContainer" containerID="9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"
Jul 07 02:48:32.739865 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.739841    2561 scope.go:117] "RemoveContainer" containerID="bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"
Jul 07 02:48:32.748492 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.748467    2561 scope.go:117] "RemoveContainer" containerID="9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"
Jul 07 02:48:32.748857 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:48:32.748829    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93\": container with ID starting with 9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93 not found: ID does not exist" containerID="9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"
Jul 07 02:48:32.748963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.748863    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93"} err="failed to get container status \"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93\": rpc error: code = NotFound desc = could not find container \"9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93\": container with ID starting with 9b51227edf828f6dbf1889eb3c9ec677f6f3121a565e8e15526930f8d52f9e93 not found: ID does not exist"
Jul 07 02:48:32.748963 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.748884    2561 scope.go:117] "RemoveContainer" containerID="bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"
Jul 07 02:48:32.749186 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:48:32.749168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39\": container with ID starting with bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39 not found: ID does not exist" containerID="bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"
Jul 07 02:48:32.749246 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.749193    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39"} err="failed to get container status \"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39\": rpc error: code = NotFound desc = could not find container \"bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39\": container with ID starting with bd104a89f45e0f5a0894cfb32e2bcb7c30b3b8b1efa11e88f65f81413abd9d39 not found: ID does not exist"
Jul 07 02:48:32.755853 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.755825    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:48:32.761299 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:32.761269    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4a8c-predictor-6c4869d48c-wg5mf"]
Jul 07 02:48:33.284160 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:33.284127    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" path="/var/lib/kubelet/pods/d21bec98-849a-41f5-8b49-622f9f6f0ea7/volumes"
Jul 07 02:48:35.724147 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:35.724121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:48:35.724538 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:35.724496    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:48:45.724910 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:45.724863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:48:55.725193 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:55.725149    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:48:58.770070 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.770026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:48:58.770535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.770325    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" containerID="cri-o://b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654" gracePeriod=30
Jul 07 02:48:58.770535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.770433    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kube-rbac-proxy" containerID="cri-o://3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79" gracePeriod=30
Jul 07 02:48:58.829969 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.829932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 02:48:58.830345 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830332    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kube-rbac-proxy"
Jul 07 02:48:58.830402 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830350    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kube-rbac-proxy"
Jul 07 02:48:58.830402 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830373    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container"
Jul 07 02:48:58.830402 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830380    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container"
Jul 07 02:48:58.830497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830440    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kserve-container"
Jul 07 02:48:58.830497 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.830450    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21bec98-849a-41f5-8b49-622f9f6f0ea7" containerName="kube-rbac-proxy"
Jul 07 02:48:58.833821 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.833796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:58.837258 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.837234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2091-kube-rbac-proxy-sar-config\""
Jul 07 02:48:58.837535 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.837518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b2091-predictor-serving-cert\""
Jul 07 02:48:58.848274 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.848243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 02:48:58.926936 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.926904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:58.927135 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.926979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:58.927135 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:58.927067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xxb\" (UniqueName: \"kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.028257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.028158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.028257 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.028216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.028496 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.028260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xxb\" (UniqueName: \"kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.028984 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.028951    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.030770 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.030751    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.045957 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.045922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xxb\" (UniqueName: \"kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb\") pod \"success-200-isvc-b2091-predictor-6c55945c5c-56pzp\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") " pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.147166 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.147127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.304134 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.304054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 02:48:59.307699 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:48:59.307662    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6781b79_010e_491b_ab3f_8786345e23b3.slice/crio-f8bff1268954392479ddeb1e278da593919549b63d98cc4aaa7bb96af0e97eb1 WatchSource:0}: Error finding container f8bff1268954392479ddeb1e278da593919549b63d98cc4aaa7bb96af0e97eb1: Status 404 returned error can't find the container with id f8bff1268954392479ddeb1e278da593919549b63d98cc4aaa7bb96af0e97eb1
Jul 07 02:48:59.857827 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.857790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerStarted","Data":"fab52f67add29ce79a67dd9446d6b517a144d8f61d2642362fb06744fa97249d"}
Jul 07 02:48:59.858343 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.857850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerStarted","Data":"104de1586c97644586beb9d514be665d5a83997092554b3889988845fcaf609f"}
Jul 07 02:48:59.858343 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.857873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerStarted","Data":"f8bff1268954392479ddeb1e278da593919549b63d98cc4aaa7bb96af0e97eb1"}
Jul 07 02:48:59.858343 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.857904    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:48:59.859714 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.859692    2561 generic.go:356] "Generic (PLEG): container finished" podID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerID="3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79" exitCode=2
Jul 07 02:48:59.859781 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.859760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerDied","Data":"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"}
Jul 07 02:48:59.888897 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:48:59.888838    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podStartSLOduration=1.888821782 podStartE2EDuration="1.888821782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:48:59.888251331 +0000 UTC m=+1851.195569757" watchObservedRunningTime="2026-07-07 02:48:59.888821782 +0000 UTC m=+1851.196140173"
Jul 07 02:49:00.864754 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:00.864721    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:49:00.866089 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:00.866060    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:01.868863 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:01.868818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:02.432066 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.432022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 07 02:49:02.436269 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.436223    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 02:49:02.523754 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.523727    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:49:02.556355 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.556325    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") pod \"e367f8d4-340b-4341-b0af-8df7a4f80c32\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") "
Jul 07 02:49:02.556572 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.556371    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h9cx\" (UniqueName: \"kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx\") pod \"e367f8d4-340b-4341-b0af-8df7a4f80c32\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") "
Jul 07 02:49:02.556572 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.556447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config\") pod \"e367f8d4-340b-4341-b0af-8df7a4f80c32\" (UID: \"e367f8d4-340b-4341-b0af-8df7a4f80c32\") "
Jul 07 02:49:02.556909 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.556883    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e367f8d4-340b-4341-b0af-8df7a4f80c32" (UID: "e367f8d4-340b-4341-b0af-8df7a4f80c32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:49:02.558585 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.558551    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e367f8d4-340b-4341-b0af-8df7a4f80c32" (UID: "e367f8d4-340b-4341-b0af-8df7a4f80c32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:49:02.558719 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.558607    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx" (OuterVolumeSpecName: "kube-api-access-7h9cx") pod "e367f8d4-340b-4341-b0af-8df7a4f80c32" (UID: "e367f8d4-340b-4341-b0af-8df7a4f80c32"). InnerVolumeSpecName "kube-api-access-7h9cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:49:02.657673 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.657569    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h9cx\" (UniqueName: \"kubernetes.io/projected/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-api-access-7h9cx\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:02.657673 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.657605    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e367f8d4-340b-4341-b0af-8df7a4f80c32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:02.657673 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.657638    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e367f8d4-340b-4341-b0af-8df7a4f80c32-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:02.875718 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.875685    2561 generic.go:356] "Generic (PLEG): container finished" podID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerID="b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654" exitCode=0
Jul 07 02:49:02.876099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.875746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"
Jul 07 02:49:02.876099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.875785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerDied","Data":"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"}
Jul 07 02:49:02.876099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.875893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv" event={"ID":"e367f8d4-340b-4341-b0af-8df7a4f80c32","Type":"ContainerDied","Data":"8990d1a74cc3993f41fbc96fba6426216d73da3d75500ab8fa2d10c2b526fd25"}
Jul 07 02:49:02.876099 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.875931    2561 scope.go:117] "RemoveContainer" containerID="3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"
Jul 07 02:49:02.885850 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.885828    2561 scope.go:117] "RemoveContainer" containerID="b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"
Jul 07 02:49:02.894459 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.894438    2561 scope.go:117] "RemoveContainer" containerID="3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"
Jul 07 02:49:02.894797 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:49:02.894778    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79\": container with ID starting with 3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79 not found: ID does not exist" containerID="3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"
Jul 07 02:49:02.894860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.894809    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79"} err="failed to get container status \"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79\": rpc error: code = NotFound desc = could not find container \"3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79\": container with ID starting with 3259b9dd7a6272a756f8af300cdbcc68e69ced6d2a25201ac48845c6b0428a79 not found: ID does not exist"
Jul 07 02:49:02.894860 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.894828    2561 scope.go:117] "RemoveContainer" containerID="b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"
Jul 07 02:49:02.895105 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:49:02.895087    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654\": container with ID starting with b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654 not found: ID does not exist" containerID="b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"
Jul 07 02:49:02.895169 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.895118    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654"} err="failed to get container status \"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654\": rpc error: code = NotFound desc = could not find container \"b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654\": container with ID starting with b730757512b768a698219528bc0fefdf7376a1d4ae14d451e0142ff20d57a654 not found: ID does not exist"
Jul 07 02:49:02.897791 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.897767    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:49:02.901718 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:02.901696    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c238-predictor-54d4799ddc-wjsvv"]
Jul 07 02:49:03.284525 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:03.284488    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" path="/var/lib/kubelet/pods/e367f8d4-340b-4341-b0af-8df7a4f80c32/volumes"
Jul 07 02:49:05.724669 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:05.724610    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:49:06.873196 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:06.873168    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:49:06.873943 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:06.873886    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:15.724692 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:15.724649    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 02:49:16.873922 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:16.873879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:25.725560 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:25.725522    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:49:26.873736 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:26.873692    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:36.873812 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:36.873771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 07 02:49:46.874810 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:46.874776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 02:49:49.222058 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.222023    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:49:49.222480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.222251    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container" containerID="cri-o://d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3" gracePeriod=30
Jul 07 02:49:49.222480 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.222306    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kube-rbac-proxy" containerID="cri-o://dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef" gracePeriod=30
Jul 07 02:49:49.313962 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.313926    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:49:49.314308 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314294    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kube-rbac-proxy"
Jul 07 02:49:49.314308 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314308    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kube-rbac-proxy"
Jul 07 02:49:49.314425 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314320    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container"
Jul 07 02:49:49.314425 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314326    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container"
Jul 07 02:49:49.314425 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314396    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kserve-container"
Jul 07 02:49:49.314425 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.314405    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e367f8d4-340b-4341-b0af-8df7a4f80c32" containerName="kube-rbac-proxy"
Jul 07 02:49:49.317564 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.317548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.320119 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.320088    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8f974-kube-rbac-proxy-sar-config\""
Jul 07 02:49:49.320265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.320181    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8f974-predictor-serving-cert\""
Jul 07 02:49:49.328422 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.328399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:49:49.376488 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.376457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26bp\" (UniqueName: \"kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.376689 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.376510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.376689 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.376562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.477276 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.477195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26bp\" (UniqueName: \"kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.477276 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.477246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.477490 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.477283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.478001 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.477977    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.480047 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.480024    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.499893 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.499857    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26bp\" (UniqueName: \"kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp\") pod \"success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") " pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.629505 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.629467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:49.779964 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:49.779929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:49:49.782255 ip-10-0-135-57 kubenswrapper[2561]: W0707 02:49:49.782224    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90e1d16_c558_4466_be42_16ab9fd74e88.slice/crio-7d9cca421e80bd47ae50f73406a3b7b774bc90af9355ad9bd197f3f91fbb6aa9 WatchSource:0}: Error finding container 7d9cca421e80bd47ae50f73406a3b7b774bc90af9355ad9bd197f3f91fbb6aa9: Status 404 returned error can't find the container with id 7d9cca421e80bd47ae50f73406a3b7b774bc90af9355ad9bd197f3f91fbb6aa9
Jul 07 02:49:50.081265 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.081232    2561 generic.go:356] "Generic (PLEG): container finished" podID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerID="dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef" exitCode=2
Jul 07 02:49:50.081485 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.081305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerDied","Data":"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"}
Jul 07 02:49:50.083050 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.083030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerStarted","Data":"159d0931e35417fbc73e72eff8c24c66571032593883c3f61f8623589da63fbc"}
Jul 07 02:49:50.083182 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.083072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerStarted","Data":"b03c354fb1725968cef8ce85aea0f4fa9304a4a755be1e3af8708dee495e7102"}
Jul 07 02:49:50.083182 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.083098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerStarted","Data":"7d9cca421e80bd47ae50f73406a3b7b774bc90af9355ad9bd197f3f91fbb6aa9"}
Jul 07 02:49:50.083182 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.083123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:50.083182 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.083150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:50.084577 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.084553    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:49:50.102848 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.102800    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podStartSLOduration=1.102778468 podStartE2EDuration="1.102778468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:49:50.101118654 +0000 UTC m=+1901.408437046" watchObservedRunningTime="2026-07-07 02:49:50.102778468 +0000 UTC m=+1901.410096858"
Jul 07 02:49:50.720150 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:50.720101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 07 02:49:51.092706 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:51.092654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:49:52.876484 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.876456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:49:52.912832 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.912737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls\") pod \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") "
Jul 07 02:49:52.912996 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.912858    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjp85\" (UniqueName: \"kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85\") pod \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") "
Jul 07 02:49:52.912996 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.912888    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config\") pod \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\" (UID: \"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a\") "
Jul 07 02:49:52.913359 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.913331    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" (UID: "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:49:52.915085 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.915047    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" (UID: "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:49:52.915204 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:52.915156    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85" (OuterVolumeSpecName: "kube-api-access-pjp85") pod "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" (UID: "a75f06b3-7b59-4a23-9b2e-e4c08af9b39a"). InnerVolumeSpecName "kube-api-access-pjp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:49:53.014445 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.014405    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:53.014445 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.014438    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjp85\" (UniqueName: \"kubernetes.io/projected/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-api-access-pjp85\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:53.014445 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.014448    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:49:53.099965 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.099927    2561 generic.go:356] "Generic (PLEG): container finished" podID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerID="d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3" exitCode=0
Jul 07 02:49:53.100152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.099979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"
Jul 07 02:49:53.100152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.100011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerDied","Data":"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"}
Jul 07 02:49:53.100152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.100072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4" event={"ID":"a75f06b3-7b59-4a23-9b2e-e4c08af9b39a","Type":"ContainerDied","Data":"77b7b789ba34dbb0580f4b79813a663a8be5ac5cd3977b24fad2c54898e6cb3e"}
Jul 07 02:49:53.100152 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.100099    2561 scope.go:117] "RemoveContainer" containerID="dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"
Jul 07 02:49:53.109589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.109570    2561 scope.go:117] "RemoveContainer" containerID="d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"
Jul 07 02:49:53.118774 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.118755    2561 scope.go:117] "RemoveContainer" containerID="dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"
Jul 07 02:49:53.119121 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:49:53.119100    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef\": container with ID starting with dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef not found: ID does not exist" containerID="dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"
Jul 07 02:49:53.119174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.119133    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef"} err="failed to get container status \"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef\": rpc error: code = NotFound desc = could not find container \"dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef\": container with ID starting with dac3cce1f8aabcf2d3604057a9e44c3e9d8a6a6011ca461af97a8191426ae8ef not found: ID does not exist"
Jul 07 02:49:53.119174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.119153    2561 scope.go:117] "RemoveContainer" containerID="d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"
Jul 07 02:49:53.119404 ip-10-0-135-57 kubenswrapper[2561]: E0707 02:49:53.119384    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3\": container with ID starting with d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3 not found: ID does not exist" containerID="d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"
Jul 07 02:49:53.119465 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.119413    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3"} err="failed to get container status \"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3\": rpc error: code = NotFound desc = could not find container \"d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3\": container with ID starting with d7e9d1977c6e0a26d836817b0ead951ff8e59965a33622572987c623569c37b3 not found: ID does not exist"
Jul 07 02:49:53.123002 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.122979    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:49:53.126980 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.126954    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4831c-predictor-854cc4c887-64bp4"]
Jul 07 02:49:53.284948 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:53.284869    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" path="/var/lib/kubelet/pods/a75f06b3-7b59-4a23-9b2e-e4c08af9b39a/volumes"
Jul 07 02:49:56.094887 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:56.094840    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:49:56.095451 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:49:56.095424    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:50:06.095682 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:50:06.095613    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:50:16.095919 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:50:16.095874    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:50:26.096327 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:50:26.096288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:50:36.095780 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:50:36.095747    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:53:09.312612 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:53:09.312584    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:53:09.316098 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:53:09.316079    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:58:09.341002 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:58:09.340893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:58:09.345819 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:58:09.345796    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 02:59:04.093563 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:04.093524    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:59:04.094572 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:04.094537    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" containerID="cri-o://b03c354fb1725968cef8ce85aea0f4fa9304a4a755be1e3af8708dee495e7102" gracePeriod=30
Jul 07 02:59:04.099640 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:04.095052    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kube-rbac-proxy" containerID="cri-o://159d0931e35417fbc73e72eff8c24c66571032593883c3f61f8623589da63fbc" gracePeriod=30
Jul 07 02:59:04.516378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:04.516273    2561 generic.go:356] "Generic (PLEG): container finished" podID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerID="159d0931e35417fbc73e72eff8c24c66571032593883c3f61f8623589da63fbc" exitCode=2
Jul 07 02:59:04.516378 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:04.516348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerDied","Data":"159d0931e35417fbc73e72eff8c24c66571032593883c3f61f8623589da63fbc"}
Jul 07 02:59:06.091298 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:06.091256    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 07 02:59:06.095635 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:06.095586    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 02:59:07.532579 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.532546    2561 generic.go:356] "Generic (PLEG): container finished" podID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerID="b03c354fb1725968cef8ce85aea0f4fa9304a4a755be1e3af8708dee495e7102" exitCode=0
Jul 07 02:59:07.532989 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.532597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerDied","Data":"b03c354fb1725968cef8ce85aea0f4fa9304a4a755be1e3af8708dee495e7102"}
Jul 07 02:59:07.663232 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.663201    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:59:07.784764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.784660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config\") pod \"d90e1d16-c558-4466-be42-16ab9fd74e88\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") "
Jul 07 02:59:07.784764 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.784736    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v26bp\" (UniqueName: \"kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp\") pod \"d90e1d16-c558-4466-be42-16ab9fd74e88\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") "
Jul 07 02:59:07.785005 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.784828    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls\") pod \"d90e1d16-c558-4466-be42-16ab9fd74e88\" (UID: \"d90e1d16-c558-4466-be42-16ab9fd74e88\") "
Jul 07 02:59:07.785065 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.785031    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d90e1d16-c558-4466-be42-16ab9fd74e88" (UID: "d90e1d16-c558-4466-be42-16ab9fd74e88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:59:07.787064 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.787025    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d90e1d16-c558-4466-be42-16ab9fd74e88" (UID: "d90e1d16-c558-4466-be42-16ab9fd74e88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:59:07.787064 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.787039    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp" (OuterVolumeSpecName: "kube-api-access-v26bp") pod "d90e1d16-c558-4466-be42-16ab9fd74e88" (UID: "d90e1d16-c558-4466-be42-16ab9fd74e88"). InnerVolumeSpecName "kube-api-access-v26bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:59:07.885868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.885817    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90e1d16-c558-4466-be42-16ab9fd74e88-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:59:07.885868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.885862    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:59:07.885868 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:07.885872    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26bp\" (UniqueName: \"kubernetes.io/projected/d90e1d16-c558-4466-be42-16ab9fd74e88-kube-api-access-v26bp\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 02:59:08.538174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.538140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6" event={"ID":"d90e1d16-c558-4466-be42-16ab9fd74e88","Type":"ContainerDied","Data":"7d9cca421e80bd47ae50f73406a3b7b774bc90af9355ad9bd197f3f91fbb6aa9"}
Jul 07 02:59:08.538174 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.538178    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"
Jul 07 02:59:08.538727 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.538210    2561 scope.go:117] "RemoveContainer" containerID="159d0931e35417fbc73e72eff8c24c66571032593883c3f61f8623589da63fbc"
Jul 07 02:59:08.547907 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.547888    2561 scope.go:117] "RemoveContainer" containerID="b03c354fb1725968cef8ce85aea0f4fa9304a4a755be1e3af8708dee495e7102"
Jul 07 02:59:08.560903 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.560870    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:59:08.567223 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:08.567189    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8f974-predictor-6c87b89cbf-rxkm6"]
Jul 07 02:59:09.286589 ip-10-0-135-57 kubenswrapper[2561]: I0707 02:59:09.286534    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" path="/var/lib/kubelet/pods/d90e1d16-c558-4466-be42-16ab9fd74e88/volumes"
Jul 07 03:03:09.368571 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:03:09.368445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 03:03:09.375006 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:03:09.374984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 03:06:18.186766 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:18.186682    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 03:06:18.187346 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:18.186963    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container" containerID="cri-o://104de1586c97644586beb9d514be665d5a83997092554b3889988845fcaf609f" gracePeriod=30
Jul 07 03:06:18.187346 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:18.187024    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kube-rbac-proxy" containerID="cri-o://fab52f67add29ce79a67dd9446d6b517a144d8f61d2642362fb06744fa97249d" gracePeriod=30
Jul 07 03:06:18.342353 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:18.342316    2561 generic.go:356] "Generic (PLEG): container finished" podID="c6781b79-010e-491b-ab3f-8786345e23b3" containerID="fab52f67add29ce79a67dd9446d6b517a144d8f61d2642362fb06744fa97249d" exitCode=2
Jul 07 03:06:18.342526 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:18.342385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerDied","Data":"fab52f67add29ce79a67dd9446d6b517a144d8f61d2642362fb06744fa97249d"}
Jul 07 03:06:19.023474 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023438    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z8mfr/must-gather-qspfc"]
Jul 07 03:06:19.023822 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023808    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023825    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023843    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023849    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023855    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kube-rbac-proxy"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023861    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kube-rbac-proxy"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023868    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kube-rbac-proxy"
Jul 07 03:06:19.023873 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023873    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kube-rbac-proxy"
Jul 07 03:06:19.024092 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023934    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kube-rbac-proxy"
Jul 07 03:06:19.024092 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023945    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kserve-container"
Jul 07 03:06:19.024092 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023951    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90e1d16-c558-4466-be42-16ab9fd74e88" containerName="kube-rbac-proxy"
Jul 07 03:06:19.024092 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.023957    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75f06b3-7b59-4a23-9b2e-e4c08af9b39a" containerName="kserve-container"
Jul 07 03:06:19.026988 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.026973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.029522 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.029503    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z8mfr\"/\"default-dockercfg-dmpjg\""
Jul 07 03:06:19.030537 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.030517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z8mfr\"/\"openshift-service-ca.crt\""
Jul 07 03:06:19.030691 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.030517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z8mfr\"/\"kube-root-ca.crt\""
Jul 07 03:06:19.038011 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.037985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z8mfr/must-gather-qspfc"]
Jul 07 03:06:19.088792 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.088758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.088984 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.088838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g489b\" (UniqueName: \"kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.189529 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.189479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g489b\" (UniqueName: \"kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.189992 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.189568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.189992 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.189916    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.198132 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.198080    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g489b\" (UniqueName: \"kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b\") pod \"must-gather-qspfc\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") " pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.349764 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.349728    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:19.486748 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.486717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z8mfr/must-gather-qspfc"]
Jul 07 03:06:19.490277 ip-10-0-135-57 kubenswrapper[2561]: W0707 03:06:19.490255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5f126f_bac9_4831_af8c_cf4fd34b120b.slice/crio-9f3a5855e0eb682e463bc9fc439062a02e56d3fcab6e67c2ed7fd4ad5853ab7d WatchSource:0}: Error finding container 9f3a5855e0eb682e463bc9fc439062a02e56d3fcab6e67c2ed7fd4ad5853ab7d: Status 404 returned error can't find the container with id 9f3a5855e0eb682e463bc9fc439062a02e56d3fcab6e67c2ed7fd4ad5853ab7d
Jul 07 03:06:19.498312 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:19.498283    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 03:06:20.357723 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:20.357548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8mfr/must-gather-qspfc" event={"ID":"4a5f126f-bac9-4831-af8c-cf4fd34b120b","Type":"ContainerStarted","Data":"9f3a5855e0eb682e463bc9fc439062a02e56d3fcab6e67c2ed7fd4ad5853ab7d"}
Jul 07 03:06:21.869348 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:21.869299    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 07 03:06:23.377403 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:23.377370    2561 generic.go:356] "Generic (PLEG): container finished" podID="c6781b79-010e-491b-ab3f-8786345e23b3" containerID="104de1586c97644586beb9d514be665d5a83997092554b3889988845fcaf609f" exitCode=0
Jul 07 03:06:23.377906 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:23.377477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerDied","Data":"104de1586c97644586beb9d514be665d5a83997092554b3889988845fcaf609f"}
Jul 07 03:06:25.499892 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.499853    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 03:06:25.548783 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.548744    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config\") pod \"c6781b79-010e-491b-ab3f-8786345e23b3\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") "
Jul 07 03:06:25.548959 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.548810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59xxb\" (UniqueName: \"kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb\") pod \"c6781b79-010e-491b-ab3f-8786345e23b3\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") "
Jul 07 03:06:25.548959 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.548908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls\") pod \"c6781b79-010e-491b-ab3f-8786345e23b3\" (UID: \"c6781b79-010e-491b-ab3f-8786345e23b3\") "
Jul 07 03:06:25.549182 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.549153    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6781b79-010e-491b-ab3f-8786345e23b3" (UID: "c6781b79-010e-491b-ab3f-8786345e23b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 03:06:25.551078 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.551046    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb" (OuterVolumeSpecName: "kube-api-access-59xxb") pod "c6781b79-010e-491b-ab3f-8786345e23b3" (UID: "c6781b79-010e-491b-ab3f-8786345e23b3"). InnerVolumeSpecName "kube-api-access-59xxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:06:25.551179 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.551144    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6781b79-010e-491b-ab3f-8786345e23b3" (UID: "c6781b79-010e-491b-ab3f-8786345e23b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:06:25.650048 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.650003    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6781b79-010e-491b-ab3f-8786345e23b3-proxy-tls\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 03:06:25.650048 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.650048    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6781b79-010e-491b-ab3f-8786345e23b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 03:06:25.650297 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:25.650061    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xxb\" (UniqueName: \"kubernetes.io/projected/c6781b79-010e-491b-ab3f-8786345e23b3-kube-api-access-59xxb\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 03:06:26.394440 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.394410    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"
Jul 07 03:06:26.394640 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.394409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp" event={"ID":"c6781b79-010e-491b-ab3f-8786345e23b3","Type":"ContainerDied","Data":"f8bff1268954392479ddeb1e278da593919549b63d98cc4aaa7bb96af0e97eb1"}
Jul 07 03:06:26.395600 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.394561    2561 scope.go:117] "RemoveContainer" containerID="fab52f67add29ce79a67dd9446d6b517a144d8f61d2642362fb06744fa97249d"
Jul 07 03:06:26.396486 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.396459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8mfr/must-gather-qspfc" event={"ID":"4a5f126f-bac9-4831-af8c-cf4fd34b120b","Type":"ContainerStarted","Data":"9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6"}
Jul 07 03:06:26.396593 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.396502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8mfr/must-gather-qspfc" event={"ID":"4a5f126f-bac9-4831-af8c-cf4fd34b120b","Type":"ContainerStarted","Data":"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"}
Jul 07 03:06:26.404550 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.404527    2561 scope.go:117] "RemoveContainer" containerID="104de1586c97644586beb9d514be665d5a83997092554b3889988845fcaf609f"
Jul 07 03:06:26.416417 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.416370    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z8mfr/must-gather-qspfc" podStartSLOduration=1.319388543 podStartE2EDuration="7.416354363s" totalImagesPullingTime="6.09696582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:06:26.41447794 +0000 UTC m=+2897.721796333" watchObservedRunningTime="2026-07-07 03:06:26.416354363 +0000 UTC m=+2897.723672752"
Jul 07 03:06:26.428941 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.428913    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 03:06:26.432670 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:26.432646    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2091-predictor-6c55945c5c-56pzp"]
Jul 07 03:06:27.284994 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:27.284962    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" path="/var/lib/kubelet/pods/c6781b79-010e-491b-ab3f-8786345e23b3/volumes"
Jul 07 03:06:48.503652 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:48.503541    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerID="2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e" exitCode=0
Jul 07 03:06:48.503652 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:48.503617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8mfr/must-gather-qspfc" event={"ID":"4a5f126f-bac9-4831-af8c-cf4fd34b120b","Type":"ContainerDied","Data":"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"}
Jul 07 03:06:48.504133 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:48.503986    2561 scope.go:117] "RemoveContainer" containerID="2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"
Jul 07 03:06:49.108073 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:49.108039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z8mfr_must-gather-qspfc_4a5f126f-bac9-4831-af8c-cf4fd34b120b/gather/0.log"
Jul 07 03:06:52.425791 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:52.425752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dqm7c_fb558531-20cf-4b45-a955-5f1ca96e1516/global-pull-secret-syncer/0.log"
Jul 07 03:06:52.623509 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:52.623474    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q8szf_0e00c637-d42b-4518-bd76-555484211f93/konnectivity-agent/0.log"
Jul 07 03:06:52.684412 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:52.684326    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-57.ec2.internal_9df865c4a0e91b8563f787ba1a893c2f/haproxy/0.log"
Jul 07 03:06:54.554241 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.554205    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-z8mfr/must-gather-qspfc"]
Jul 07 03:06:54.554733 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.554410    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-z8mfr/must-gather-qspfc" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="copy" containerID="cri-o://9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6" gracePeriod=2
Jul 07 03:06:54.560228 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.560201    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-z8mfr/must-gather-qspfc"]
Jul 07 03:06:54.793692 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.793670    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z8mfr_must-gather-qspfc_4a5f126f-bac9-4831-af8c-cf4fd34b120b/copy/0.log"
Jul 07 03:06:54.795413 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.795396    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:54.797641 ip-10-0-135-57 kubenswrapper[2561]: E0707 03:06:54.797603    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qspfc\" is forbidden: User \"system:node:ip-10-0-135-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z8mfr\": no relationship found between node 'ip-10-0-135-57.ec2.internal' and this object" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:54.918448 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.918420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output\") pod \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") "
Jul 07 03:06:54.918448 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.918456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g489b\" (UniqueName: \"kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b\") pod \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\" (UID: \"4a5f126f-bac9-4831-af8c-cf4fd34b120b\") "
Jul 07 03:06:54.920522 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.920496    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4a5f126f-bac9-4831-af8c-cf4fd34b120b" (UID: "4a5f126f-bac9-4831-af8c-cf4fd34b120b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:06:54.920993 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:54.920973    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b" (OuterVolumeSpecName: "kube-api-access-g489b") pod "4a5f126f-bac9-4831-af8c-cf4fd34b120b" (UID: "4a5f126f-bac9-4831-af8c-cf4fd34b120b"). InnerVolumeSpecName "kube-api-access-g489b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:06:55.019523 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.019487    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a5f126f-bac9-4831-af8c-cf4fd34b120b-must-gather-output\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 03:06:55.019523 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.019516    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g489b\" (UniqueName: \"kubernetes.io/projected/4a5f126f-bac9-4831-af8c-cf4fd34b120b-kube-api-access-g489b\") on node \"ip-10-0-135-57.ec2.internal\" DevicePath \"\""
Jul 07 03:06:55.284120 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.284031    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" path="/var/lib/kubelet/pods/4a5f126f-bac9-4831-af8c-cf4fd34b120b/volumes"
Jul 07 03:06:55.532673 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.532617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z8mfr_must-gather-qspfc_4a5f126f-bac9-4831-af8c-cf4fd34b120b/copy/0.log"
Jul 07 03:06:55.532993 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.532971    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerID="9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6" exitCode=143
Jul 07 03:06:55.533079 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.533045    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8mfr/must-gather-qspfc"
Jul 07 03:06:55.533079 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.533071    2561 scope.go:117] "RemoveContainer" containerID="9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6"
Jul 07 03:06:55.541059 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.541038    2561 scope.go:117] "RemoveContainer" containerID="2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"
Jul 07 03:06:55.556388 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.556247    2561 scope.go:117] "RemoveContainer" containerID="9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6"
Jul 07 03:06:55.556648 ip-10-0-135-57 kubenswrapper[2561]: E0707 03:06:55.556605    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6\": container with ID starting with 9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6 not found: ID does not exist" containerID="9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6"
Jul 07 03:06:55.556702 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.556652    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6"} err="failed to get container status \"9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6\": rpc error: code = NotFound desc = could not find container \"9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6\": container with ID starting with 9bd0ab2c80fb97f2ccede3a1d33436984834722a2774473a31042cc4b03c96e6 not found: ID does not exist"
Jul 07 03:06:55.556702 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.556674    2561 scope.go:117] "RemoveContainer" containerID="2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"
Jul 07 03:06:55.556923 ip-10-0-135-57 kubenswrapper[2561]: E0707 03:06:55.556904    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e\": container with ID starting with 2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e not found: ID does not exist" containerID="2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"
Jul 07 03:06:55.556973 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:55.556929    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e"} err="failed to get container status \"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e\": rpc error: code = NotFound desc = could not find container \"2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e\": container with ID starting with 2ed68d26d11be07dc7f2a775fd4bc03df02f040a021b122ea4bae423d1559e1e not found: ID does not exist"
Jul 07 03:06:56.737109 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.737080    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/alertmanager/0.log"
Jul 07 03:06:56.761438 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.761404    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/config-reloader/0.log"
Jul 07 03:06:56.785913 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.785886    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/kube-rbac-proxy-web/0.log"
Jul 07 03:06:56.809633 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.809591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/kube-rbac-proxy/0.log"
Jul 07 03:06:56.840262 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.840234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/kube-rbac-proxy-metric/0.log"
Jul 07 03:06:56.867360 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.867332    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/prom-label-proxy/0.log"
Jul 07 03:06:56.892012 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.891985    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f04d47c4-c2ba-4d44-9be2-2aefb4592436/init-config-reloader/0.log"
Jul 07 03:06:56.975937 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.975843    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dmsjx_b8789e80-7a85-4faf-be6a-12a4bfeb2491/kube-state-metrics/0.log"
Jul 07 03:06:56.995501 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:56.995471    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dmsjx_b8789e80-7a85-4faf-be6a-12a4bfeb2491/kube-rbac-proxy-main/0.log"
Jul 07 03:06:57.018856 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.018824    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dmsjx_b8789e80-7a85-4faf-be6a-12a4bfeb2491/kube-rbac-proxy-self/0.log"
Jul 07 03:06:57.054468 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.054439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f7d695f6f-5txkx_f1c00182-20c8-4aec-ab43-749702f26fed/metrics-server/0.log"
Jul 07 03:06:57.096459 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.096427    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-8qmf5_0cca002c-ca33-4534-96a1-418df6fa1b13/monitoring-plugin/0.log"
Jul 07 03:06:57.133168 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.133121    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqpwp_2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a/node-exporter/0.log"
Jul 07 03:06:57.155084 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.155053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqpwp_2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a/kube-rbac-proxy/0.log"
Jul 07 03:06:57.186498 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.186474    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqpwp_2f7b5a23-4df7-4b1c-ae5c-3ac32c3ad22a/init-textfile/0.log"
Jul 07 03:06:57.403263 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.403233    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z8nbk_8312a6ec-7348-4770-baec-9bab8b97eb2f/kube-rbac-proxy-main/0.log"
Jul 07 03:06:57.426492 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.426452    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z8nbk_8312a6ec-7348-4770-baec-9bab8b97eb2f/kube-rbac-proxy-self/0.log"
Jul 07 03:06:57.449448 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.449419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z8nbk_8312a6ec-7348-4770-baec-9bab8b97eb2f/openshift-state-metrics/0.log"
Jul 07 03:06:57.501917 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.501877    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/prometheus/0.log"
Jul 07 03:06:57.521870 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.521845    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/config-reloader/0.log"
Jul 07 03:06:57.544032 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.544006    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/thanos-sidecar/0.log"
Jul 07 03:06:57.569909 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.569885    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/kube-rbac-proxy-web/0.log"
Jul 07 03:06:57.600163 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.600140    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/kube-rbac-proxy/0.log"
Jul 07 03:06:57.637960 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.637933    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/kube-rbac-proxy-thanos/0.log"
Jul 07 03:06:57.674716 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.674690    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa0679bb-9a22-43f2-97eb-e9bd0556624a/init-config-reloader/0.log"
Jul 07 03:06:57.849065 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.849036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694b9757b-864c7_e2a84723-1983-477b-8546-fe81d2fc88fd/telemeter-client/0.log"
Jul 07 03:06:57.889906 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.889871    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694b9757b-864c7_e2a84723-1983-477b-8546-fe81d2fc88fd/reload/0.log"
Jul 07 03:06:57.933884 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.933805    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694b9757b-864c7_e2a84723-1983-477b-8546-fe81d2fc88fd/kube-rbac-proxy/0.log"
Jul 07 03:06:57.981339 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:57.981301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/thanos-query/0.log"
Jul 07 03:06:58.018810 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:58.018787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/kube-rbac-proxy-web/0.log"
Jul 07 03:06:58.059836 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:58.059808    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/kube-rbac-proxy/0.log"
Jul 07 03:06:58.104852 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:58.104823    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/prom-label-proxy/0.log"
Jul 07 03:06:58.145893 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:58.145870    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/kube-rbac-proxy-rules/0.log"
Jul 07 03:06:58.187934 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:06:58.187851    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66d5b8bbf6-dfk87_dbf87aa6-80d0-4e04-b674-3345f5a61a64/kube-rbac-proxy-metrics/0.log"
Jul 07 03:07:00.308575 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.308544    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-745b8bd5f6-rk2wc_8eb437d2-e558-4e41-8ffd-21a80878e9f2/console/0.log"
Jul 07 03:07:00.849252 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849215    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"]
Jul 07 03:07:00.849568 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849556    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kube-rbac-proxy"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849570    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kube-rbac-proxy"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849582    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="gather"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849588    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="gather"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849594    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="copy"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849599    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="copy"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849612    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container"
Jul 07 03:07:00.849617 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849617    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container"
Jul 07 03:07:00.849848 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849690    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="gather"
Jul 07 03:07:00.849848 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849698    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kube-rbac-proxy"
Jul 07 03:07:00.849848 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849704    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5f126f-bac9-4831-af8c-cf4fd34b120b" containerName="copy"
Jul 07 03:07:00.849848 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.849712    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6781b79-010e-491b-ab3f-8786345e23b3" containerName="kserve-container"
Jul 07 03:07:00.853946 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.853927    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:00.856694 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.856669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4pjz\"/\"kube-root-ca.crt\""
Jul 07 03:07:00.856891 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.856688    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4pjz\"/\"openshift-service-ca.crt\""
Jul 07 03:07:00.856891 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.856740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h4pjz\"/\"default-dockercfg-5lngh\""
Jul 07 03:07:00.862017 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.861992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"]
Jul 07 03:07:00.975142 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.975106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-podres\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:00.975327 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.975159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-proc\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:00.975327 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.975267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-sys\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:00.975327 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.975297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-lib-modules\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:00.975327 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:00.975324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znz8x\" (UniqueName: \"kubernetes.io/projected/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-kube-api-access-znz8x\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076676 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-sys\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076676 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-lib-modules\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znz8x\" (UniqueName: \"kubernetes.io/projected/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-kube-api-access-znz8x\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-podres\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076747    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-sys\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-proc\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076825    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-proc\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076861    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-lib-modules\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.076929 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.076884    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-podres\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.085581 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.085555    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znz8x\" (UniqueName: \"kubernetes.io/projected/07b24c0a-0cd5-47ba-9cd5-62da60ddc040-kube-api-access-znz8x\") pod \"perf-node-gather-daemonset-dd6fh\" (UID: \"07b24c0a-0cd5-47ba-9cd5-62da60ddc040\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.165169 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.165071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.299820 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.299695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"]
Jul 07 03:07:01.302615 ip-10-0-135-57 kubenswrapper[2561]: W0707 03:07:01.302586    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07b24c0a_0cd5_47ba_9cd5_62da60ddc040.slice/crio-acbbafaf9ac780a0c376ab63820ba32c0908eb937963c8dd1bc1cafabb60914a WatchSource:0}: Error finding container acbbafaf9ac780a0c376ab63820ba32c0908eb937963c8dd1bc1cafabb60914a: Status 404 returned error can't find the container with id acbbafaf9ac780a0c376ab63820ba32c0908eb937963c8dd1bc1cafabb60914a
Jul 07 03:07:01.558587 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.558488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh" event={"ID":"07b24c0a-0cd5-47ba-9cd5-62da60ddc040","Type":"ContainerStarted","Data":"40582df0c2fd5032901d1a89b9beea3ea57c1f126077828b88042ce9c82033ed"}
Jul 07 03:07:01.558587 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.558548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh" event={"ID":"07b24c0a-0cd5-47ba-9cd5-62da60ddc040","Type":"ContainerStarted","Data":"acbbafaf9ac780a0c376ab63820ba32c0908eb937963c8dd1bc1cafabb60914a"}
Jul 07 03:07:01.558587 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.558577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:01.577293 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.577238    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh" podStartSLOduration=1.5772211569999999 podStartE2EDuration="1.577221157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 03:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:07:01.575989274 +0000 UTC m=+2932.883307665" watchObservedRunningTime="2026-07-07 03:07:01.577221157 +0000 UTC m=+2932.884539547"
Jul 07 03:07:01.881238 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.881193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtz8x_f2c85221-43cb-4ca4-94b1-86a895e8c648/dns/0.log"
Jul 07 03:07:01.905233 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:01.905186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtz8x_f2c85221-43cb-4ca4-94b1-86a895e8c648/kube-rbac-proxy/0.log"
Jul 07 03:07:02.013254 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:02.013218    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t2mlv_ff16fd4f-d7f3-4591-a4fe-8b8905692255/dns-node-resolver/0.log"
Jul 07 03:07:02.584357 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:02.584315    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ngcnf_d816096e-154e-490a-9c94-815e35733b2f/node-ca/0.log"
Jul 07 03:07:03.822793 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:03.822762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rlhqp_ec88dc48-87af-434b-b851-19eb2e8d449f/serve-healthcheck-canary/0.log"
Jul 07 03:07:04.200470 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:04.200367    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlvjt_0823e9dd-da0a-4311-b643-5519fcf57046/kube-rbac-proxy/0.log"
Jul 07 03:07:04.222937 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:04.222912    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlvjt_0823e9dd-da0a-4311-b643-5519fcf57046/exporter/0.log"
Jul 07 03:07:04.245675 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:04.245645    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlvjt_0823e9dd-da0a-4311-b643-5519fcf57046/extractor/0.log"
Jul 07 03:07:06.508435 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:06.508410    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-4n7f4_8fc1061f-bd4d-418b-a3d4-03457fb9a16d/server/0.log"
Jul 07 03:07:06.673109 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:06.673076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-np8c8_043105df-4a88-4c68-9f42-3f92c756fd69/manager/0.log"
Jul 07 03:07:06.716520 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:06.716483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cfgcw_e69cbb55-84ad-4e5a-bd7a-f8643981f5e6/s3-init/0.log"
Jul 07 03:07:06.732749 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:06.732720    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cfgcw_e69cbb55-84ad-4e5a-bd7a-f8643981f5e6/download-hf-model/0.log"
Jul 07 03:07:06.762158 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:06.762069    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2ck2p_c17eef29-ca4b-4558-9de2-28f0b21d18fb/seaweedfs/0.log"
Jul 07 03:07:07.573296 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:07.573271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-dd6fh"
Jul 07 03:07:12.178122 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.178094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/kube-multus-additional-cni-plugins/0.log"
Jul 07 03:07:12.202923 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.202844    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/egress-router-binary-copy/0.log"
Jul 07 03:07:12.227792 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.227763    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/cni-plugins/0.log"
Jul 07 03:07:12.250609 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.250582    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/bond-cni-plugin/0.log"
Jul 07 03:07:12.271192 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.271160    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/routeoverride-cni/0.log"
Jul 07 03:07:12.293662 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.293617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/whereabouts-cni-bincopy/0.log"
Jul 07 03:07:12.314638 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.314608    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6jngg_d3172c72-6087-4af8-bf4f-24475348408a/whereabouts-cni/0.log"
Jul 07 03:07:12.688117 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.688087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gqwgh_49f46c00-1bba-4491-aa68-aef5b6d3c490/kube-multus/0.log"
Jul 07 03:07:12.844861 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.844823    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t5bd7_ed4f47c0-1a2e-4045-826a-65ba949859d3/network-metrics-daemon/0.log"
Jul 07 03:07:12.864610 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:12.864574    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t5bd7_ed4f47c0-1a2e-4045-826a-65ba949859d3/kube-rbac-proxy/0.log"
Jul 07 03:07:13.620116 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.620089    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-controller/0.log"
Jul 07 03:07:13.640112 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.640080    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/0.log"
Jul 07 03:07:13.640112 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.640102    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovn-acl-logging/1.log"
Jul 07 03:07:13.658756 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.658720    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/kube-rbac-proxy-node/0.log"
Jul 07 03:07:13.680094 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.680060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 03:07:13.702174 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.702144    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/northd/0.log"
Jul 07 03:07:13.724765 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.724741    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/nbdb/0.log"
Jul 07 03:07:13.750423 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.750394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/sbdb/0.log"
Jul 07 03:07:13.815898 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:13.815866    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9q4km_4756d2f8-25d8-497e-be3e-d934a593ccf4/ovnkube-controller/0.log"
Jul 07 03:07:15.298033 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:15.298003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r2x82_d1ddd8ce-320b-4115-a3b5-d7082fe3da1a/network-check-target-container/0.log"
Jul 07 03:07:16.241880 ip-10-0-135-57 kubenswrapper[2561]: I0707 03:07:16.241848    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wdnwc_67ea23be-df66-49b6-8ebc-efb851af5659/iptables-alerter/0.log"